The CHALLENGES I Faced When Working With GCP Vertex AI API!

Introduction:

This week’s blog post reflects on another part of my dissertation, the integration with Google Cloud Platform’s Vertex AI API.

This was arguably the most complicated part of development and was an entirely new experience for me.

This blog post aims to summarise the process of using this technology, highlighting the challenges faced and ultimately, the success of working with Vertex AI API.

As always, before we get into this week’s article, be sure to applaud, comment and share with your friends.

Happy Reading!

Challenges:

Challenge 1: Working with the Node JS SDK:

The Node JS examples didn’t directly correlate with my Next JS proposed solution. Admittedly, I should just accept this as something that will always happen with every single project I work on.

This led to me scrapping the SDK, in favour of the API. In my opinion, this is actually better practise as the SDK is just a wrapper for the API (performing the same functionality) however this led to further challenges.

I had to work with cURL and Postman before achieving the functionality programmatically and there were several challenges.

Challenge 2: Setting up a ‘GCP Service Account’

While there is documentation out there on how to do this, it was an entirely new experience with me. From past experience with AWS, I’ve found setting up accounts to be a real painpoint although to be fair, it felt simpler with GCP.

Challenge 3: Sending a Request to the API (Permissions):

This builds upon ‘Challenge 1’ as the API requests required an access token. With the cURL request, this was simple to provide as it was generated each time a request was made through the terminal.

Programmatically, I was unable to find a way to generate an access token therefore to satisfy this functionality, I was manually generating a token through the GCP CLI. Ultimately, this is one limitation with my dissertation project and many hours was spent, attempting to address it.

For the reasons stated above, this feature works offline in a development environment however it’s not complete in the sense of a ‘production ready’ feature.

Challenge 4: Working with the API’s response:

After the previous struggles of attempting to gain a successful API response (200), I finally got one!

Unfortunately, the struggles were not done there as I’d made the rather naïve assumption of assuming that the response would be in JSON. This resulted in a large amount of time spent attempting to access the response as JSON when in reality, this was not possible.

Upon realising, I wasn’t working with JSON, after a bit of investigating, I thought that I was working with a response in the format of a ‘Server-Side Event’. Again, lots of time was spent and I eventually realised that this too wasn’t the actual response format.

After even more investigation, it turned out that the response was in the format of ‘NDJSON’.

I’d never heard of this before and there was no compatible NPM library for converting from NDJSON to JSON (this would’ve saved so much time).

Thankfully, the format is fairly close to JSON. It still took many hours but eventually, I was able to split the NDJSON based on ‘\n’ characters and then parse each line into JSON. This then allowed me to use the API’s response within my Next JS project.

Challenge 5: API Costs:

Unfortunately, because I was passing in all of my stringified data (as part of the API request), this resulted in a large request body which began to incur costs.

Due to limited time, I couldn’t really address this. I did try to create a Cloud Function which would shutdown all resources when a budget was hit however this was incomplete.

To address this in future, it would require smarter thinking about minimising the size of a request body, while still providing enough data in order to generate an accurate response.

Conclusion:

Despite all of the challenges, I liked working with GCP Vertex AI API and it was particularly satisfying to implement.

The fact that this API works in a similar nature to Chat GPT’s API made it particularly appealing.

Final Things:

As always, thank you for taking the time to read this article.

All my links are here.