GIDS Conference 2024 - Takeaways on 25th April

Overview

The Great International Developer Summit (GIDS) 2024 took place from 23rd to 26th April at IISc Bangalore, featuring over 150 tech talks with some well-known speakers. Sponsored by leading tech firms, including D.E. Shaw India, my workplace, the conference provided a fantastic learning experience. I had a blast learning new concepts and networking with the speakers when I attended on the 25th and 26th.

The talks covered various parts of Software Development, but in line with the current market trend, many were focused on AI (especially Generative AI). This blog post documents my takeaways and insights (Whatever I remember or have written down) from the talks I attended on 25th April, 2024. I’ll make a separate post for the talks on April 26th.

All of the talks will be uploaded to SaltMarch on 17th May 2024. I’ll link here once it’s up. Speaker details can be found in the abstracts I’ve linked.

Talks

Software Architecture for Gen AI - Rebecca Parsons

Abstract: https://developersummit.com/session/software-architecture-for-gen-ai

Rebecca started off Day 3 of the conference with an incredible talk giving a high level overview of the different elements involved in LLM models and architectures of applications that leverage these models. The intent was to broadly cover the different elements rather than deep diving into anything specific.

1. Most organizations should leverage models and tools (such as langchain) built by the big players in this space rather than building their own libraries and models. The cost, data, effort, and energy requirements are enormous and impractical for the majority of the firms out there. Try to see if a problem has been solved by other firms before implementing something of your own.
2. 2023 was the year of Proof of Concepts (PoC) where firms rushed to try out LLMs for a wide variety of use case. 2024 is the year of productionizing those PoCs, but it’s proving to be much more difficult than we anticipated. They do not meet non-technical user expectations and you cannot explain to them why the model is behaving in a certain way.
3. Common Generative AI problems such as hallucination, inappropriate responses, and non-determinism are not bugs, but rather a feature of the model. It’s a result of what makes LLMs so powerful. There is a need for fitness functions to validate and test a model for specific use cases. My Java brain thinks of them functionally equivalent as unit tests in TDD.
4. Retrieval Augumented Generation (RAG) is currently the most common and simplest way of passing domain specific context to an LLM model without re-training it. Edge AI is another idea for low latency and privacy focused applications where some parts of the generation happens on-device rather than cloud.
5. The context lengths of LLM models have been increasing exponentially (now reaching 200K tokens) over the past 2 years, but we might soon hit a roadblock as we don’t have enough data to train, despite the PBs of data that internet generates. One idea to bridge this data gap is to use LLMs to generate data for future LLMs to train on, but this is questionable and loses originality.

GraalVM in a nutshell - Sachin Pikle

Abstract: https://developersummit.com/session/graalvm-in-a-nutshell

In this 30 min session, Sachin gave an overview of the ahead-of-Time native image compilation in GraalVM. GraalVM is an alternative to OpenJDK (again created by Oracle) with some interesting optimizations.

1. GraalVM has two modes of executing a Java application,
   1. Just-In-Time (JIT) compiler, which is similar to standard OpenJDK, but has a different compiler optimization logic. I assume the performance would be similar when averaged across different use cases.
   2. Ahead-Of-Time (AOT) compiler, which generates a native executable of the application allowing it to be run anywhere without a JVM.
2. An AOT binary is much smaller in size than the equivalent JIT application since it doesn’t need to bundle unused classes, dynamic code caches, metaspace class data, etc. It is considered safer because it doesn’t allow loading unknown classes or dynamic libraries at run time. AOT executables are not portable since they are created for a specific platform.
3. Ahead-of-time compilation excels in application start-up time performance. However, the builds are slower, which is expected because the compiler performs all the optimizations and code compilation during the build phase.
4. As per demos shared by Sachin, the application throughput and latency do not differ noticeably between an OpenJDK compiled application and a GraalVM AOT compiled application. However, GraalVM had a significantly better application start up time and initial memory usage.
5. This compiler makes the most sense in serverless architectures such as Lambda functions where start up time is crucial even if portability is sacrificed.

I will have to test this out, but for long running applications with varied use cases, I believe AOT will do worse in performance since JIT can make dynamic code optimizations during run time, while AOT doesn’t have that flexibility.

Real-Time Bidding at Scale - Deepanshi Dhamija

Abstract: https://developersummit.com/session/real-time-bidding-at-scale-a-deep-dive-into-sub-40ms-strategies

Deepanshi focused on the Adobe AdTech business where she covered the various elements involved when a Demand-Side Platform (DSP) makes a bid to the Ad exchange servers. The cut-throat business of DSPs and Ad exchanges requires sub-100ms latencies to win ad slots and grab eye balls. I learnt more about Ad Tech in this 30 min talk than I’ve ever known before.

1. An Ad Exchange server (e.g., Google AdX) is a marketplace which connects buyers and sellers. Sellers are people who visit a website with ad spaces, where each ad space is real estate to be bid on by the advertisers. Visitors are connected to the Ad Exchange through a Supply Side Platform (which is not very relevant for this discussion). The buyers are DSPs (such as Adobe Advertising) who buy these ad spaces and show their ads, usually from different organizations. Ad Exchange intermediates this transaction and decides which DSP should win a given ad space.
2. The problem statement for DSPs then is to choose which ad slots to buy, buy them at a reasonable price, and still outbid competitors, while doing this as quickly as possible usually under 100ms.
3. When a bid request is made from the Ad Exchange to the Adobe DSP, there are many steps of filtering that take place to make a smart bid. The DSP decides which advertisement to show in an Ad slot depending on the users’ demographics and browser cookies. If the ad doesn’t match, no bid is made. Otherwise, a bid amount is decided by an ML model trained on past data.
4. Despite this space seeming very different than what I’ve worked on, the same core principles continue to apply,
   1. Deploy the DSPs close to users (and ad exchanges) to reduce latency.
   2. Keep the most useful (and mostly static) data in memory for quick access.
   3. If the data is large, cache it.
5. One interesting idea is to use a sidecar container to refresh and store the caches. This will be an independent service (microservice?) that stores cached data and can be queried by other services for fast retrieval.

Total ReDoS: Dangers of Regex in JS - Phil Nash

Abstract: https://developersummit.com/session/total-redos-the-dangers-of-regex-in-javascript

This was a really fun talk by Phil about the dangers of Denial of Service caused by inefficient regular expressions. Most of the talk is based on one of his articles which I’d recommend checking out!

1. The JavaScript engine uses a process called backtracking to retry when a string match fails. For e.g., take the regular expression .+b and input string “aaaaabcd”, (Source Link)
   • The engine will match .+ (one or more of any chars) with string “aaaaabc” and fail when matching b from the regex.
   • It backtracks a bit and tries again by matching “aaaaab” with .+ and fails when matching “cd” with b.
   • It backtracks again to “aaaaa” with .+ and finally succeeds now since “bcd” matches with b.
2. Backtracking can easily lead to polynomial time complexity which behaves poorly with large strings, especially when no match exists. This can partially be solved by limiting the input length or maximum length that should be matched by the regex.
3. However, we can accidentally write regular expressions that cause Catastrophic Backtracking which have an exponential time complexity, trigerring downtimes for much smaller strings. One such example is .*.*=.*, which is well explained in this cloudflare analysis.
4. If it’s possible, avoid using regular expressions, you don’t need regular expressions to validate e-mails and domain addresses. Use in-built or library functions. Validate your regular expressions using tools such as regex101 or regexper which visualizes the expression as a flow chart.
5. Integrate static code analysis tools such as this ESlint Plugin (eslint-plugin-regexp) or Sonarqube to catch backtracking issues early on. Setup alerts on dependabot to be alerted when a dependency you’re using has a backtracking vulnerability.

From SQL to NoSQL with Redis - Brian Sam-Bodden

Abstract: https://developersummit.com/session/from-sql-to-nosql-with-redis

Brian started off the talk with an overview of Redis (REmote DIctionary Server) and discussed how Redis OM Spring can be used to do SQL-like searching on a Redis DB in a Spring application.

1. Relational DBs emerged as the first type of databases during the initial days of the internet when the use cases were much simpler. The schema of these DBs us the golden source and applications are required to follow it. However, after the boom of the internet, there was a need for more flexible DBs which led to the creation of NoSQL databases where the applications decide the DB schema based on the data it stores.
2. Redis, which initially started as a cache server, is a key-value database supporting a variety of data types. Values can be strings, JSON objects, sets, maps, hashes, and even hyperloglog structures. Redis offers indexing functionality on complex data structures and full text search capabilities for strings. NoSQL when combined with powerful search capabilities effectively mimics the functionalities of SQL.
3. Redis OM Spring is an ORM for Spring to define Redis objects to persist and perform advanced search, similar to Hibernate for relational databases. Most of the remaining talk revolved around using this library, for which I’d recommend going through this README if you’re interested.

Creating Async applications with Virtual Threads - Venkat Subramaniam

Abstract: https://developersummit.com/session/creating-asynchronous-applications-with-virtual-threads

Venkat gave a brilliant talk on the problems with Java physical threads, how Kotlin solves it, and how Java 19’s virtual threads can be used instead.

1. Ask how many threads you should create, not how many threads you can create. Physical threads come with their own memory overheads and unecessary parallelism can often degrade performance. The formula below is a good way to estimate how many physical threads you need,

   \(\displaylines{No. \: of \: Threads <= {No. \: of \: cores \over \left(1 - Blocking \: Factor \right)}\\ where, 0 <= Blocking \: Factor < 1\\ Blocking \: Factor \propto IO \: Intensitivity}\)

   • Blocking Factor is higher for an I/O Intensive application.
   • If the application spends 20% of the time stuck in I/O, blocking factor should be 0.2.
   • If the application spends 80% of the time stuck in I/0, blocking factor should be 0.8.
2. Kotlin coroutines are light weight logical threads that can be suspended in one physical thread and resumed in another. This is made possible by a Continuation data structure in Kotlin that remembers the current state of the task and restores it for execution.
3. Historically, Java threads have always been physical threads. We can define a thread pool using Executors.newFixedThreadPool and specify the number of threads to initialize with. Any task running on the executor service will always take up one physical thread until they complete, even if the task is blocked on some I/O.
4. With Java 19, the introduction of Executors.newVirtualThreadPerTaskExecutor allows for the creation of a virtual thread for every task submitted to the executor service. Virtual Threads are logical threads that are mounted over a physical thread when execution is required. They can be suspended and resumed on a different physical threads, and there’s no need to specify the number of virtual threads to create, as a new one is created for each task with minimal overhead.
5. Virtual threads are suitable for I/O intensive tasks that spend most of the time blocked. For compute intensive tasks, it’s not recommended to use virtual threads since they don’t improve the throughput and might even worsen it due to overheads.

12 Essential Practices in Continuous Delivery - Venkat Subramaniam

Abstract: https://developersummit.com/session/twelve-essential-practices-to-be-effective-in-continuous-delivery

Despite being the last session of the day (at 6 p.m.), this talk stood out as my favorite of the entire conference because I resonated with many of the points that Venkat made. This session presented a strong (and funny) argument for Continuous Delivery in software development. I highly recommend checking out the full video once it’s available.

1. The point of Continuous Delivery is to be able to deploy a feature to Production at any time without fear of the disrupting the application. Continuous Integration is the prerequisite, and Continuous Delivery is the goal. CD doesn’t necessarily eliminate human effort, but rather aims to reduce it to the bare minimum (such as approvals). Above all, following Continuous Delivery requires discipline.
2. The DRY (Don’t Repeat Yourself) principle is about the duplication of effort, not duplication of code. If you perform repeated manual testing or manual deployments, you are violating DRY. Continuous Delivery streamlines processes and reduces the risk of errors by automating various stages of the testing and acceptance pipeline.
   • Write and maintain automated unit, acceptance, and integration tests.
   • Use fakes/mocks whenever necessary for end-to-end testing.
   • Always use automated deployment scripts.
3. The quality of code is inversely proportional to the effort it takes to understand it. Keep functions small (follow SRP) and add unit tests. Be kind to your fellow teammates :)
4. Automate your database updates; otherwise you could be blocked on DBA for days on end. This is often not followed and kills your CD dreams.
5. Ask yourself: If I don’t come in to work today, how does it affect the company and its bottomline? You might have to introspect if the answer is not satisfactory.