I am currently collaborating with other developers on a site. We use a private github repo and want to configure our dev server so that a push to the repo will trigger an action that deploys the latest source code to our development server.

My first question is can anyone recommend a good approach to deploying source from github to one's server when someone pushes changes?. I found this approach which seems a little suspect to me. I'm guessing it will be necessary to use github's hooks, but the approach in oodavid's README doesn't specify an https url for the deployment script, doesn't seem to supply or check for any credentials, refers to a var $PWD which is not defined in the script (is it supposed to come from $_GET?), and seems totally vulnerable to exploitation by malicious hackers who could bog your server down by simply requesting it over and over again.

My other question relates to file permissions. Seems to me that having my entire web application writable by Apache seems like a gaping security problem.

Assuming I find some way to deploy code using a github hook that doesn't require apache write permissions on my entire project, what should my file permissions be? The other collaborators and I all need to be able to write each and every file in the web application (as does our git-deployment operation) and apache may need to write a few directories as well. Seems to me that there might be some clever group-based approach possible here, but it seems like quite a puzzle. Seems to me we'll also need to set up some kind of umask, which is not a concept I understand all that well.

I apologize if these questions seem simple. I'm under a considerable strain right now and don't have a lot of time. I was hoping that if someone has already tackled this problem that they might share some of their findings.

    I have been using Fabric for deployment of Django web projects for some time. In my simple use case, it is just a way for me to bundle logging into the server and running a bunch of commands to pull from the designated repository, migrate database, copy static files, reload webserver, etc, into a single command.

    I imagine that you could use the same approach. Unfortunately, git does not seem to have a post-push hook such that you could run a Fabric command automatically after a push to the designated branch on the GitHub repository, so you would either need to modify this approach to do the git push in the Fabric command then remember to run this Fabric command instead of the git push, or remember to run it after the git push.

    This is inferior to getting the push event to trigger the deployment separately since each developer with push permissions to the GitHub repository also needs SSH access to the server and must remain connected to the server throughout the deployment, but it will sidestep your issue in that if you and your collaborators have no permission problems doing this manually, then there will be no permission problems automating with Fabric.

      Oooh interesting. I've no experience with Python at all, but expect I might be able to sort things out.

      Am I to understand that this is a lot like apple remote desktop in that you can execute a single command on many remote hosts without having to actually ssh into each machine? If so, that sounds quite powerful. Can you comment on the security of fabric? I'm guessing the power it brings the tradeoff of security concerns and one must be very responsible.

        sneakyimp wrote:

        Am I to understand that this is a lot like apple remote desktop in that you can execute a single command on many remote hosts without having to actually ssh into each machine?

        Well, you are using SSH: Fabric does so on your behalf. But yes, by specifying multiple hosts you can execute the same set of commands on each of them running a single Fabric command.

        sneakyimp wrote:

        Can you comment on the security of fabric?

        It is as secure of your SSH setup, i.e., no passwords stored in the fabfile, but rather you can use key authentication as you would with your usual SSH agent and client. You can also read the docs on SSH behavior. One thing to note is that in order to access the GitHub repository using your credentials, you would need SSH agent forwarding (e.g., by defining the setting in ~/.ssh/config and then passing --forward-agent as an argument to fab).

          OK so I have managed to look into this deeper and I believe that I've devised a way to use github's webhooks to automatically refresh my dev server's repo when someone pushes changes to the github repo. There are a few steps:
          1) Establish a command-line script to pull the latest files from the github repo to the dev server and possibly set permissions correctly. Minimally, this would be something like 'git pull.' I'm still wondering how to have my server provide authentication credentials to pull from the private repo, but I'll get around to that.
          2) add an entry to the sudoers file that gives the apache user permission to execute just the one script described in step 1. This should be more secure than giving apache write permission on my entire web app directory as long apache doesn't have write permission to the command-line script itself. Some helpful links.
          3) Create a url on our website (i.e., a controller and method in CodeIgniter in my case) to receive and authenticate the hook request generated by github using a secret key and hash scheme. If authentication passes, execute the git-fetch-and-set-permissions script described in step 1 using [man]exec[/man] or something like that.
          4) Configure github with the correct url in step 3 and a secret key shared by the script to validate incoming POST requests.

          I still have some PHP-related questions about how to authenticate incoming hook payloads from github. (See docs on hook payloads.
          The push event payload comes in and we must calculate an HMAC digest calculation on it. This payload doesn't look like key-value pairs but rather something like raw post data. Where in PHP does one find the raw post data these days?
          Docs say "HTTP requests made to your webhook’s configured URL endpoint will contain several special headers:

          Header             Description
          X-Github-Event     Name of the event that triggered this delivery.
          X-Hub-Signature    HMAC hex digest of the payload, using the hook’s secret as the key (if configured).
          X-Github-Delivery  Unique ID for this delivery.
          

          Where does one find these headers? In $SERVER? $ENV?

            Derokorian;11045721 wrote:

            Request headers are in $SERVER.


            I don't think they are. I had a function:

            	public function hook() {
            		file_put_contents("/tmp/_POST", serialize($_POST));
            		file_put_contents("/tmp/_SERVER", serialize($_ENV));
            		file_put_contents("/tmp/raw_post_input", file_get_contents('php://input'));
            	} // hook()
            

            the contents of _SERVER was simply this:

            a:0:{}

            But the request that was sent from github was this:

            Request URL: http://example.com/git/hook
            Request method: POST
            content-type: application/json
            Expect: 
            User-Agent: GitHub-Hookshot/2d21501
            X-GitHub-Delivery: 57d20c00-c2da-11e4-9902-532bc1f9529d
            X-GitHub-Event: ping
            X-Hub-Signature: sha1=74ebc28a4a3e1b6e06f059c27f12a75d2d9ec727
            
            {
              "zen": "Responsive is better than fast.",
              "hook_id": 1234567,
              "hook": {
                "url": "https://api.github.com/repos/someuser/test/hooks/1234567",
                "test_url": "https://api.github.com/repos/someuser/test/hooks/1234567/test",
                "ping_url": "https://api.github.com/repos/someuser/test/hooks/1234567/pings",
                "id": 1234567,
                "name": "web",
                "active": true,
                "events": [
                  "push"
                ],
                "config": {
                  "url": "http://example.com/git/hook",
                  "content_type": "json",
                  "insecure_ssl": "0",
                  "secret": "********"
                },
                "last_response": {
                  "code": null,
                  "status": "unused",
                  "message": null
                },
                "updated_at": "2015-03-05T01:52:56Z",
                "created_at": "2015-03-05T01:52:56Z"
              },
              "repository": {
                "id": 31684148,
                "name": "test",
                "full_name": "someuser/test",
                "owner": {
                  "login": "someuser",
                  "id": 7654321,
                  "avatar_url": "https://avatars.githubusercontent.com/u/7654321?v=3",
                  "gravatar_id": "",
                  "url": "https://api.github.com/users/someuser",
                  "html_url": "https://github.com/someuser",
                  "followers_url": "https://api.github.com/users/someuser/followers",
                  "following_url": "https://api.github.com/users/someuser/following{/other_user}",
                  "gists_url": "https://api.github.com/users/someuser/gists{/gist_id}",
                  "starred_url": "https://api.github.com/users/someuser/starred{/owner}{/repo}",
                  "subscriptions_url": "https://api.github.com/users/someuser/subscriptions",
                  "organizations_url": "https://api.github.com/users/someuser/orgs",
                  "repos_url": "https://api.github.com/users/someuser/repos",
                  "events_url": "https://api.github.com/users/someuser/events{/privacy}",
                  "received_events_url": "https://api.github.com/users/someuser/received_events",
                  "type": "User",
                  "site_admin": false
                },
                "private": false,
                "html_url": "https://github.com/someuser/test",
                "description": "trying to test github deployment on push",
                "fork": false,
                "url": "https://api.github.com/repos/someuser/test",
                "forks_url": "https://api.github.com/repos/someuser/test/forks",
                "keys_url": "https://api.github.com/repos/someuser/test/keys{/key_id}",
                "collaborators_url": "https://api.github.com/repos/someuser/test/collaborators{/collaborator}",
                "teams_url": "https://api.github.com/repos/someuser/test/teams",
                "hooks_url": "https://api.github.com/repos/someuser/test/hooks",
                "issue_events_url": "https://api.github.com/repos/someuser/test/issues/events{/number}",
                "events_url": "https://api.github.com/repos/someuser/test/events",
                "assignees_url": "https://api.github.com/repos/someuser/test/assignees{/user}",
                "branches_url": "https://api.github.com/repos/someuser/test/branches{/branch}",
                "tags_url": "https://api.github.com/repos/someuser/test/tags",
                "blobs_url": "https://api.github.com/repos/someuser/test/git/blobs{/sha}",
                "git_tags_url": "https://api.github.com/repos/someuser/test/git/tags{/sha}",
                "git_refs_url": "https://api.github.com/repos/someuser/test/git/refs{/sha}",
                "trees_url": "https://api.github.com/repos/someuser/test/git/trees{/sha}",
                "statuses_url": "https://api.github.com/repos/someuser/test/statuses/{sha}",
                "languages_url": "https://api.github.com/repos/someuser/test/languages",
                "stargazers_url": "https://api.github.com/repos/someuser/test/stargazers",
                "contributors_url": "https://api.github.com/repos/someuser/test/contributors",
                "subscribers_url": "https://api.github.com/repos/someuser/test/subscribers",
                "subscription_url": "https://api.github.com/repos/someuser/test/subscription",
                "commits_url": "https://api.github.com/repos/someuser/test/commits{/sha}",
                "git_commits_url": "https://api.github.com/repos/someuser/test/git/commits{/sha}",
                "comments_url": "https://api.github.com/repos/someuser/test/comments{/number}",
                "issue_comment_url": "https://api.github.com/repos/someuser/test/issues/comments{/number}",
                "contents_url": "https://api.github.com/repos/someuser/test/contents/{+path}",
                "compare_url": "https://api.github.com/repos/someuser/test/compare/{base}...{head}",
                "merges_url": "https://api.github.com/repos/someuser/test/merges",
                "archive_url": "https://api.github.com/repos/someuser/test/{archive_format}{/ref}",
                "downloads_url": "https://api.github.com/repos/someuser/test/downloads",
                "issues_url": "https://api.github.com/repos/someuser/test/issues{/number}",
                "pulls_url": "https://api.github.com/repos/someuser/test/pulls{/number}",
                "milestones_url": "https://api.github.com/repos/someuser/test/milestones{/number}",
                "notifications_url": "https://api.github.com/repos/someuser/test/notifications{?since,all,participating}",
                "labels_url": "https://api.github.com/repos/someuser/test/labels{/name}",
                "releases_url": "https://api.github.com/repos/someuser/test/releases{/id}",
                "created_at": "2015-03-04T22:50:29Z",
                "updated_at": "2015-03-04T22:50:29Z",
                "pushed_at": "2015-03-04T22:50:30Z",
                "git_url": "git://github.com/someuser/test.git",
                "ssh_url": "git@github.com:someuser/test.git",
                "clone_url": "https://github.com/someuser/test.git",
                "svn_url": "https://github.com/someuser/test",
                "homepage": null,
                "size": 0,
                "stargazers_count": 0,
                "watchers_count": 0,
                "language": null,
                "has_issues": true,
                "has_downloads": true,
                "has_wiki": true,
                "has_pages": false,
                "forks_count": 0,
                "mirror_url": null,
                "open_issues_count": 0,
                "forks": 0,
                "open_issues": 0,
                "watchers": 0,
                "default_branch": "master"
              },
              "sender": {
                "login": "someuser",
                "id": 7654321,
                "avatar_url": "https://avatars.githubusercontent.com/u/7654321?v=3",
                "gravatar_id": "",
                "url": "https://api.github.com/users/someuser",
                "html_url": "https://github.com/someuser",
                "followers_url": "https://api.github.com/users/someuser/followers",
                "following_url": "https://api.github.com/users/someuser/following{/other_user}",
                "gists_url": "https://api.github.com/users/someuser/gists{/gist_id}",
                "starred_url": "https://api.github.com/users/someuser/starred{/owner}{/repo}",
                "subscriptions_url": "https://api.github.com/users/someuser/subscriptions",
                "organizations_url": "https://api.github.com/users/someuser/orgs",
                "repos_url": "https://api.github.com/users/someuser/repos",
                "events_url": "https://api.github.com/users/someuser/events{/privacy}",
                "received_events_url": "https://api.github.com/users/someuser/received_events",
                "type": "User",
                "site_admin": false
              }
            }

              I'm an idiot. this line obviously wrong:

              file_put_contents("/tmp/_SERVER", serialize($_ENV)); 

              I corrected it

              file_put_contents("/tmp/_SERVER", serialize($_SERVER)); 

              kinda weird how the header keys get transformed:

              ["HTTP_X_GITHUB_EVENT"]=>
                string(4) "ping"
                ["HTTP_X_GITHUB_DELIVERY"]=>
                string(36) "57d20c00-c2da-11e4-9902-532bc1f9529d"
                ["CONTENT_TYPE"]=>
                string(16) "application/json"
                ["HTTP_X_HUB_SIGNATURE"]=>
                string(45) "sha1=74ebc28a4a3e1b6e06f059c27f12a75d2d9ec727"

                To prevent namespace collisions. Otherwise someone could send you a request that included headers named "Document-Root" or suchlike; or alternatively an existing server variable from PHP could shadow a legitimate HTTP header in a later version of the protocol.

                  Weedpacket;11045731 wrote:

                  To prevent namespace collisions. Otherwise someone could send you a request that included headers named "Document-Root" or suchlike; or alternatively an existing server variable from PHP could shadow a legitimate HTTP header in a later version of the protocol.

                  Ah yes. Now I feel like you've mentioned this before.

                  So I have set up a webhook on github. It's possible to do this via browser. Still wondering a couple of things.

                  What's the best way to get raw post data? This is currently working pretty well for me (see hash question below) but I'm wondering if this is a good way to get raw post data or not?

                  $raw_post_data = file_get_contents('php://input');
                  

                  The signature looks like this:

                  sha1=74ebc28a4a3e1b6e06f059c27f12a75d2d9ec727

                  I've got [man]hash_hmac[/man] calculating a matching sha1 hash from the raw post data, but I'm wondering whether it's safe to take whatever's before the = and feed it to hash_hmac as the name of an algorithm or not. If not safe, would the correct approach be to just check if the specified hash string is a member of the array returned by [man]hash_algos[/man]?

                    sneakyimp wrote:

                    I've got [man]hash_hmac[/man] calculating a matching sha1 hash from the raw post data, but I'm wondering whether it's safe to take whatever's before the = and feed it to hash_hmac as the name of an algorithm or not. If not safe, would the correct approach be to just check if the specified hash string is a member of the array returned by [man]hash_algos[/man]?

                    A quick check of the PHP manual shows that hash_hmac returns false when the algorithm is unknown hence checking with hash_algos separately is unnecessary as long as you check the return value of hash_hmac.

                      laserlight;11045813 wrote:

                      A quick check of the PHP manual shows that hash_hmac returns false when the algorithm is unknown hence checking with hash_algos separately is unnecessary as long as you check the return value of hash_hmac.

                      Given that the hash algorithm comes directly from user input, I was concerned about what might happen if someone were to attempt a buffer overflow exploit or something. Seems to me that checking the supplied value against hash_algos() is a good idea so I am implementing it that way.

                      Can someone tell me why we need the function [man]hash_equals[/man] ? I don't really understand how timing could affect a comparison of two strings.

                        OK another stumbling block.

                        I've got code in place to receive github's POST operation when someone pushes to the repo and I am able to authenticate these posts to make sure they are in fact legit. However, I am having trouble figuring out how to get apache to perform a git pull operation on a private repo because github will prompt for a username/password. I see that github offers deploy key functionality to grant access to a particular repo, but I'm having a bit of trouble sorting out how to get this set up on my server. It apparently involves generating a key pair for apache. https://gist.github.com/oodavid/1809044This tutorial suggests putting the key in /var/www -- but this seems like a bad idea to me. Isn't that directory the default web root for apache if a request doesn't match any virtual hosts? I'm also wondering whether to look on this key pair as THE key pair for apache or whether there is some way to limit a key to simply being used to pull from git.

                          sneakyimp;11045805 wrote:

                          What's the best way to get raw post data? This is currently working pretty well for me (see hash question below) but I'm wondering if this is a good way to get raw post data or not?

                          $raw_post_data = file_get_contents('php://input');
                          

                          Yes

                            However, I am having trouble figuring out how to get apache to perform a git pull operation on a private repo because github will prompt for a username/password. I see that github offers deploy key functionality to grant access to a particular repo, but I'm having a bit of trouble sorting out how to get this set up on my server. It apparently involves generating a key pair for apache.

                            This doesn't even make sense to me, apache is a webserver. It would not be doing git pulls at all as far as I know? Am I missing something??

                            but this seems like a bad idea to me. Isn't that directory the default web root for apache if a request doesn't match any virtual hosts?

                            Only if its configured that way in httpd.conf. There is nothing set in stone that says an unmatched request goes here, in fact you can make unmatched requests return an error code if you prefer!

                              sneakyimp wrote:

                              Man, this seems unnecessarily complicated:
                              https://alvinabad.wordpress.com/2013...e-git-command/

                              Complicated? Because he lists two options, perhaps.

                              As a sysadmin, I wouldn't call that too complex. As a programmer without sysadmin experience ... I'm not sure. But you're smart enough to handle that, I've no doubt. I'm dumb as a brick most days and it doesn't look too difficult ;-)

                              It's probably just that the project as a whole is long? I heard that. Some days I just wanna, I dunno ... something not too healthy, I'd imagine...

                                Thanks for verifying that, Derokorian.

                                The problem I'm grappling with now is authenticating with github. The git-pull command doesn't let you specify a private key to authenticate with so one must find another, more complicated way to make sure that apache provides its private key when needed. There seem to be a few options, the most simple of which seems to be to generate a key pair for apache in the default location (see here). Unfortunately, this location appears to be /var/www and I'm concerned (perhaps wrongly) about apache's private key somehow being exposed to web requests. This section of the apache config seems a bit worrisome:

                                #
                                # Relax access to content within /var/www.
                                # 
                                <Directory "/var/www">
                                    AllowOverride None
                                    # Allow open access:
                                    Require all granted
                                </Directory>
                                

                                Sadly, all the other ways of getting the private key supplied seem complicated (see this):
                                Use the GIT_SSH environment variable
                                Use a wrapper script

                                  Derokorian;11045829 wrote:

                                  This doesn't even make sense to me, apache is a webserver. It would not be doing git pulls at all as far as I know? Am I missing something??

                                  Re-read the first post in this thread. Apparently it's apache's responsbility to do the git-pull command in response to a url request. This is how githubs webooks work.

                                  Derokorian;11045829 wrote:

                                  Only if its configured that way in httpd.conf. There is nothing set in stone that says an unmatched request goes here, in fact you can make unmatched requests return an error code if you prefer!

                                  I am mostly baffled by apache virtual host directives and that sort of thing. I've scanned all the config files and I posted what I was most concerned about. I also have this vague recollection that apache's webroot defaults to /var/www (at least on Ubuntu?) and if you don't define any virtual hosts then that's where it will go to try and server web requests. I've experienced this problem when I somehow request a web server by IP instead of by some hostname.

                                    dalecosp;11045833 wrote:

                                    Complicated? Because he lists two options, perhaps.

                                    It's complicated because he lists two options, both of which require the creation of scripts, the setting of permissions, and apparently the utilization of some mystical daemon called ssh-agent (also described here). I'm sure I could accomplish one of these other options once, but this project has in the past jumped from server to server to server repeatedly and these approaches contribute a fair amount of extra server configuration work that must be replicated on the other machines -- if I even manage to remember how.

                                    dalecosp;11045833 wrote:

                                    As a sysadmin, I wouldn't call that too complex. As a programmer without sysadmin experience ... I'm not sure. But you're smart enough to handle that, I've no doubt. I'm dumb as a brick most days and it doesn't look too difficult ;-)

                                    I appreciate very much your vote of confidence, but plumbers and carpenters have been swarming my apartment since last Wednesday trying to repair damage from a burst pipe. I'm about to lose my mind over here.

                                    dalecosp;11045833 wrote:

                                    It's probably just that the project as a whole is long? I heard that. Some days I just wanna, I dunno ... something not too healthy, I'd imagine...

                                    As I mentioned above, solving this problem once is probably manageable. It's repeating that solution whenever we jump servers that I'm worried about.