TLDR: You may be using ContainerDirectory without a HostDirectory or you may need to update the 03build.sh to build with the --no-cache=true flag.
After a bazillion hours later, I finally fixed this with my use case. I am using CodePipeline to run CodeCommit, CodeBuild, and Elastic Beanstalk to create a continuous integration / continuous delivery solution in AWS with docker. The issue I ran into was the CodeBuild was successfully building and publishing new docker images to AWS ECR (EC2 container registry) and EBS was correctly pulling down the new image, but yet the docker image was never getting updated on the server.
After inspecting the entire process of how EBS builds the docker image (there's a really great article here, part 1 and here part 2 that gives an overview), I discovered the issue.
To add to the article, there is a 3-stage process in EBS on the EC2 instances that are spun-up for deploying docker images.
This 3-stage process is a sequence of bash files that are executed which are located in /opt/elasticbeanstalk/hooks/appdeploy/.
The pre-stage contain the following shell scripts:
00clean_dir.sh - Cleans directory where source will be downloaded, removes docker containers and images (e.g. cleanup)
01unzip.sh - Downloads source from S3 and unzips it
03build.sh - This is where the magic happens where EC2 will build your docker image from your Dockerfile or Dockerrun.aws.json. After much testing, I realized this build script was building my updated image but I modified this script to include the --no-cache flag on docker build as well.
The enact stage is where my caching issue was actually occurring. The enact stage consists of:
00run.sh - this is where docker run is executed against the image that was generated in the pre stage based on environment variables and settings in your Dockerrun.aws.json. This is what was causing the caching issue for me.
01flip.sh - Converts from aws-staging to current-app and a lot of other stuff
When I would execute docker run from the image that was generated in Pre stage, 03build.sh, I would see my updated changes. However, when I would execute the 00run.sh shell script, the old changes would appear. After investigating the docker run command, it was executing
`Docker command: docker run -d -v null:/usr/share/nginx/html/ -v /var/log/eb-docker/containers/eb-current-app:/var/log/nginx ca491178d076`
The -v null:/usr/share/nginx/html/ is what was breaking it and causing it not to update. This was because my Dockerrun.aws.json file had
without a referenced host location. As a result, any future changes I made, didn't get updated.
For my solution, I just removed the "Volumes" array as all of my files are contained in the docker image I upload to ECR. Note: You may need to add the --no-cache to the 03build.sh as well.