3.3 Process Requests

Now that we have a secure web book, we can finally start processing our requests. This means that we are going to enumerate over our commits and we are going to process the added, modified and removed collections from each of those commits. And really we can process our commits in reversed order. We can take the latest commit, process it, and if there are any files that we have already processed we can just ignore them in the earlier commits because we already have the latest version, there's no sense in updating those files. So the first this we should do is get our commits, but in reverse order. So let's say our commits = and we want payload.Commits. We want to OrderByDescending. And we're going to do so based upon the time stamp. So that will give us our commits. And we just want to enumerate over them. So, var commits in commits. And then we want to process that commit. Now we could have a service that would accept the individual commit and then inside of that it will process the three collections. And that is probably what I am going to do because that means that there is less code inside of the controller and eventually, I also want to hide as much of this as possible. I mean this is all very important stuff, but we could hide all of this inside of an extension method for requests. So that might be something that we do. Later on in this lesson. We'll just see if we get to that. Okay, we want to process our commit. These are commits for the push event. Let's say that we have a push payload processor, something along those lines. We'll just call this variable processor we will new up, get hub, push processor. And then we would have a method called a process that we would then pass the commit to. And, we don't need to create this processor inside of the loop, we need to do that outside so that we can have code that looks like that. So, I like the way that this looks. Let's just go ahead and create this. So this can be inside of our services. And we will create that class. And I don't think this method needs to return anything, so we will just have public. This probably should be asynchronous. So Task and Process. The reason why this should be async is because we will be downloading the files and we want to do that asynchronously. So we have a commit, which we want to pull in here. We also need that using statement for our model's namespace. And now we are good to go. Although this should be process async, because it is an asynchronous method. As far as we are concerned, there's not much difference between a file that was added and one that was modified. Because regardless, we have to retrieve it and save it. So really we can simplify our code if we combine those two things together. So let's say that we have a variable called filesToRetrieve and that will be comit and then added. We will use the Union extension method and we will union with the Modified. And then we will look over these files to retrieve, so var file in filesToRetrieve and then we want to download them. So we need a method to do that. So let's come down here, let's make this private. This is going to be asynchronous, and it's going to return a task of string, and we'll say GetFile Async. And we need the url of the file that we want to retrieve. And then this is going to be fairly straightforward. We create a new HttpClient which we need a using statement for system net Http. So that we can then download that file. So we will simply return await client.getstring Async and that URL. Now, the next thing that we need to do is build the URL that we are going to pass to the get file async method. Now the paths that are in the Jason Payload are relative to the repository. If we look at that payload very quickly we can see that that is the case. So, this is part of the url. We just need the rest of it, the host and all of that stuff, and we are going to download the raw data. That way we don't have to mess with any APIs or anything like that. So let's go to GitHub and we will look at what that Base URL Is going to be. And I have Fiddler running and that can cause a problem with https requests. So I will close Fiddler, refresh, and there we go. Now I don't have anything inside of the gitpowered repository. I'm saving that for when we finally get our application up and running. And it's out there so that github can actually access it. So for the development of this application I have this hook repo and that is where those payloads that I have provided Did and the code download come from. You can see that there is the posts folder. There's the 2014 and 2015 folders. And then there are files inside of those folders. Now we want the raw data. So let's go to one of those files and let's click on raw. And we can get the base part of that url. We want everything up until the posts. So we will take that part of the url, we will come back to our code and we will put that, let's put that right here inside of the ProcessAsync method. Now, this is something that we would normally put inside of our configuration. We wouldn't want this hardcoded inside of our code. So that is something that we will need to address later on. But for right now, this is going to work. Let's have a variable called a base URL, and we are going to set that to that URL. And then inside of the 4H loop, we are going to create our URL, and it is going to have two parts. The first is the base. So we will have baseUrl there. Then we will have our /file. Then we want to download that file. So, lets say, var data, we want to await, GetFileasync, pass in that url. and that will give us our data. Now we need to keep track of our files that we process. That way that whenever we come across another file that we have already processed, we can just skip it. Let's add a private, this can be a read only list of string that we will call this underscore processed and let's go ahead and new up that constructor and inside of the for each loop, we'll say if_processed contains and then we'll pass in file. Then we will just continue. Because in that case, there's nothing to process because we have already processed it. But if our file is not in _processed, then we will process it and then the last thing we will do Is add that file to that collection. Now everything else that we need to do is related to the file system, we want to save these files, and then we also need to process the removed files, and remove those. So let's go ahead and write that loop, var file and commit and removed. And then, we will need to remove those. Now, the file system is a data store. So we could use a repository pattern to create or update or remove those files. That way, if we ever decide to change how we store this information, then we could just swap out the repository. Now that would mean that we would need to implement an interface. And in something more than a personal project, yeah we would probably want to do that. But for the sake of simplicity, we're just going to stick with a concrete class. Let's say that we have a private, read only field. Yeah, let's make it read only. Let's call this file store, and we'll call it underscore files, and then inside of our constructor, which we are now going to have, we will create that file store. But we also need to know where we want to store those files. Now if we look at solution explore then we can get an idea of what would make sense. It really would make sense to do so inside of the www root. We could add a folder here called posts or whatever it is we're going to be working with and store those there. So we need to get the path to wwwroot. Now in order to do that we have to go back to our controller. And we will use NVC's six's build in dependency injection to get an object that will then allow us to get the path to that www root folder. So, let's first of all have a private read only field. It's going to be a type IHostingEnvironment. And we will need a using statement here for Microsoft.AspNet.Hosting. And let's just called this _environment. And then we will need a constructor that is going to accept one of these ihosting environment objects. And we will rely upon NVC6's IOS container to give us that object. So, the constructor is PushWebhookController, we want that IHostingEnvironment, let's just call it environment. And then inside of the constructor, we will set our private field equal to the parameter and that will give us what we need. Now this I hosting environment object has a property called web root path that will give us the path to that www root. Whenever we create our GitHub processor, we could pass in that web root path folder. Then, the inside of our processor, we could then set that path to the constructor of FileStore for the base path. So let's call this wwwrootPath and then we will pass that on to FileStore. And then inside of our loops after we have retrieved the contents of the file, we can say, underscore files dot and then, create or update and then, we would pass in the file name because that would, essentially be the identifier and then we would need the data for that file. And then, for removing, we would have underscore files and then we could have remove or delete, guess delete would be better then we just pass in this file name. So now we just need to create this file store class and this is more data related so I"m going to create a new folder cold data, and that is where I will put that new class. So I'm going to call it file store, we need two methods we need the create or update. Now both of these need to be async actually, so let's do public async task, and that first is create or update, but since this is asynchronous we need async. We have the file name. We also need the data of that file, so there is that. And then we also need public async task and delete. And then we just need the file name. Now we need a constructor here so we can store the base path. So let's first of all have that. And private readonly string under store. Base path. And then our constructor will have the base path as the parameter. Then we will set our underscore base path equal to base path. Now let's start with the create or update method. The first thing we need to do is combine our paths together. So let's say our path equals and then we will have our _basepath followed by our slash, although I think it's the other slash and then our file name. Now it's possible that the folders inside of filename, we have posts and then 2015 or 2014 and then the name of that file. It's possible that those folders don't exist. So we first of all need to ensure that those directories are there. If they're not, then we want to create them. So let's create another variable. We'll just call this folder path. And we're going to cheat a little bit. We're going to take the path. We're going to remove, or actually, let's do replace. And we are going to take the file name, and replace it with an empty string. That will give us. The full path to the folder so that we can then check that so, let's say, real file name equals path and we need the using segment here for system I.O. We want the get file name method and then, we will pass in path that will give us the file name which would be the file name.txt then we will replace the real file name with an empty string so that that will give us our directories. Then we can check to see if that directory exists. So if not, directory exists, we pass in the folder path. And, if it doesn't, then we will create it. So, directory, create directory, folder path. And then that will allow us to write the file or update the file. So, we will create a new stream writer. And we'll do so with File.CreateText. Now we can't do new StreamWriter. The reason because the DNX core doesn't have a StreamWriter constructor that accepts a string. So if we tried to compile this and pass in the string for the path, then we would get an error. So instead, we're going to say, File.CreateText. We will pass in the path to our file. And then we will simply write that file. So, we will use the WriteAsync method. We pass in data. And that is really that we need to do there for CreateOrUpdateAsync. Now deleting a file is going to be much easier. We do need to build the path. So we will do the same thing there. And then we will check to see if that file exists because if it doesn't, well then there's no need to do anything else. So we'll say File.Exists we will pass in a path and then we will just File.Delete and we will pass in the path and that will complete that, although the name of the delete methods needs to be Async. Now our code doesn't compile as is because we need to add some using statements and change the names of these methods. First of all let's add a using statement for our data name space for our FileStore. And we also need to change the names of these methods. We have CreateOrUpdate Update asynch and then we have delete asynch. We also need to await these. Because they are asynchronous. And that should be all that we need to do here inside of our GitHub processor class. And so now let's go to our controller. And we need to add a using statement for that, for our service's namespace. And we need to await our process async method. Now we're not going to test this in this lesson because we're also going to do some more cleanup. We're also going to go ahead and add the things that we need to add to our configuration. So, let's first of all rebuild this, make sure that it compiles because that's all that I care about right now, and it does. So, in the next lesson, we're going to take our github secret key. We're also going to take that url and put that in our app settings dot jason file. We're going to write some code so that we can retrieve those values. And we will also perform some cleanup.