Ok, so why was I overcomplicating things?
It stemmed from three things…
1 – We weren’t sure if the app that needed to communicate with the CRM had the abilty to securely store variables. (I now know that it can).
2 – The app that needed to make the the API call had to do this synchronously, and a delay wasn’t going to work – though a default condition if the API call failed was OK. It was the wait that was a no go.
3 – This was a new process for us, so we had no idea what concurrent load would look like.
So that means it was time to overthink!
Just kidding. I’ve read that if you have the luxury of time its always best to try to identify a few solutions and then rule them out. That’s basically what happened here, and as I prototyped each the limitations of one stood out, the other had bad code smell – why was I doing so much, and well the third worked as detailed in part 1.
The first solution to kick around was using Zoho Flow as a catch for the API request from this app. It was effectively a sort of API gateway, but all it really was is a Zoho Flow initiated by a webhook trigger. Two issues with this.
1 – Zoho Flow was going to act on a webhook trigger, go do its business with the CRM and then respond. If this could be wired up (there wasn’t certainty the app initiating the call could deal with the response), it was starting to get complicated. You can do certain things in Zoho Flow – its a typical drag and drop workflow engine, and you can do more complex things in Zoho Flow Custom functions, but if you wanted to do anything more complicated you need to jump to the CRM via a Flow custom function and then create a CRM function. We’re starting to chain a lot of stuff together. If you recall we wanted to make an API call with a value, do a lookup and return a value. This was too heavy for something simple like that.
2 – Well, not to be outdone the next plan was even worse. I’ll detail that in a moment, but the TLDR is what if I couldn’t secure the API key in the originating app? This was something that still needed some research and I was looking for any option that provided more security. One option was to expose the Zoho CRM function via OAuth and then only store a refresh token in the originting app.
More detail on the second option…
I still had the same constraints – mainly being this was a synchronous call and I had no idea what concurrent load could look like. I was mostly OK having a refresh token in that originating app – so now what?
The plan kind of looked like this:
– After I made my CRM function, enabled REST and then generated an acccess and refresh token pair for the right scopes I needed a way for the other app to use it.
– First thought was have the other app use the refresh token and a dummy access token to request a current access token.
– I’d have another Zoho CRM function that would use some org variables in the CRM to keep track of what was happening. Things like the current access token, the expiration time, and a few other things.
– From here, the function basically would take in the refresh token from the other app. It get the most recent access token and its expriation from the org vars.
– If the access token had expired, use the refresh token to generate a new access token and send that back – otherwise send the current one back.
– Then the originating application had what it needed to make the next CRM function call to actually do the work.
So that worked, but it was getting thick. Then a discussion with the stakeholder is when it was revealed we no idea about concurrent load – so if we had 10 requests per second what happens when the access token expired and was being refreshed while another processes was asking for it? Would we have a race condition, how would it fail? Nothing more complication couldn’t fix.
The next bolt on was to make a resource lock so another process would know that the access token was being refreshed and to wait it out a few seconds. Deluge doesn’t have a loop construct or sleep, so I had to do some hacky thing:
if(accesstoken_refresh_lock == "true")
{
iterations = {0,1,2};
for each iteration in iterations
{
try
{
response = invokeurl
[
url :"https://postman-echo.com/delay/1"
type :GET
];
}
catch (ex)
{
info "Error waiting - " + ex.toString();
}
// Check again
accesstoken_refresh_lock = zoho.crm.getOrgVariable("accesstoken_refresh_lock");
if(accesstoken_refresh_lock == "false")
{
// Lock is clear, we can proceed
break;
}
}
}
Some internet sleuthing gave me the tricks here – iterate over a list a few times, and call some durable service to fake a sleep. Then check again. In our case this was going to be acceptable. If the API failed, there was a default condition the originating app would run with, but it was going to be a better customer experience to have this synchronous call to get a real value.
After setting the lock it was a matter of getting a refreshed access token, updating the org vars with the new token and expiration and then returning it.
This worked too.
So now the chain looked like:
originating app -> get an access token from CRM function -> call CRM function -> get return value and continue
Not complicated itself, but there was a mess making it happen.
So, as you might have read in part 1 – eventually it was discovered that the originating app could securely store variables, and I ultimately went with the last option detailed in part 1. In the end it was an interesting journey and I learned a bit more than expected on these fringe areas of Deluge and APIs I wasn’t dealing with often. All in all good learning, and that’s a great outcome!