“How GitHub’s Copilot Chat Enables Developers to Easily Seek Code Assistance”

Earlier this year, GitHub rolled out Copilot Chat, a ChatGPT-like programming-centric chatbot, for organizations subscribed to Copilot for Business. Copilot Chat has been a highly anticipated feature for businesses and organizations, since its beta launch. According to Shuyin Zhao, VP of product management at GitHub, Copilot Chat has become the most widely adopted AI developer tool in history. However, the launch was only limited to organizations, but now, Copilot Chat is finally available to individual Copilot customers as well.

As of today, Copilot Chat can be found in the sidebar in Microsoft’s IDEs, Visual Studio Code and Visual Studio, included as a part of GitHub Copilot paid tiers and free for verified teachers, students and maintainers of certain open source projects.

“As home to the world’s developers, we’ve brought to market what is now the most widely adopted AI developer tool in history,” Shuyin Zhao, VP of product management at GitHub, told TechCrunch in an email interview. “And code complete was just the beginning.”

Little else about Copilot Chat has changed since the beta. The chatbot is still powered by GPT-4, OpenAI’s flagship generative AI model, fine-tuned specifically for dev scenarios. Developers can prompt Copilot Chat in natural language to get real-time guidance, for example asking Copilot Chat to explain concepts, detect vulnerabilities or write unit tests.

Like all generative AI models, the model underpinning Copilot Chat, GPT-4, was trained on publicly available data — some of which is copyrighted or under a restrictive license. Vendors including GitHub argue fair use doctrine shields them from copyright claims. However, this has not stopped coders from filing class action lawsuits against GitHub, Microsoft (GiHub’s parent company), and OpenAI over what they allege are open source licensing and IP violations.

I asked Shuyin Zhao whether codebase owners will have a chance to opt out of training, now, in the event that they wish to do so. She said that there’s no new mechanism for this with the broader launch of Copilot Chat.

• She instead suggested that codebase owners make their repositories private to prevent them from being included in future training sets.

I have to imagine codebase owners won’t take too kindly to that suggestion — there’s many reasons for keeping copyrighted code public, least of which crowdsourcing bug hunting. However, GitHub is evidently not willing to budge on training data opt-outs — for now, at least.

Generative AI models, including GPT-4, also have a tendency to hallucinate, or confidently make up facts — which is particularly problematic in the coding realm. According to a recent Stanford study, developers who use AI assistants to code tend to produce code that’s less secure compared to those who don’t use AI assistants, in part because the AI assistants introduce buggy or deprecated code snippets.

Shuyin Zhao said that GPT-4 performs “better” against hallucinations compared to the older model that once powered Copilot and pointed to exploit-mitigating features such as filters for insecure code patterns, which notify Copilot Chat users of vulnerabilities like hardcoded credentials, SQL injections and path injections. However, she stressed the importance of close human review of any AI-suggested code.

“GitHub Copilot is powered by OpenAI’s models, which we’ve found to be the best models for the services we offer today,” Shuyin Zhao said. “We’re in a really strong position to continue empowering developers with the AI tools they need to build better, more secure software at scale — and to have fun while they’re doing it.”

In October, Microsoft CEO Satya Nadella told analysts that Copilot had 1 million paying users and ~37,000 enterprise clients. However, it’s incumbent on GitHub to make Copilot even more attractive lest it lose ground to competitors and, for that matter, lose cash.

According to a Wall Street Journal piece, Copilot loses an average of $20 a month per user, with some customers costing GitHub as much as $80 a month. The high price of running the underlying AI models is reportedly to blame — a problem gen AI coding startup Kite ran into, forced it to shut down early last December.

As GitHub struggles to make Copilot profitable, Amazon continues upgrading CodeWhisperer, perhaps Copilot’s best-resourced rival.

In April, Amazon made CodeWhisperer free of charge to developers without any usage restrictions. That month also saw the launch of CodeWhisperer Professional Tier, which added single sign-on with AWS Identity and Access Management integration as well as higher limits on scanning for security vulnerabilities. An enterprise plan for CodeWhisperer launched in September. And in early November, Amazon “optimized” CodeWhisperer to provide “enhanced” suggestions for app development on MongoDB, the open source database management program.

Aside from CodeWhisperer, Copilot has competition in startups like Magic, Tabnine, Codegen and Laredo as well as open source models like Meta’s Code Llama and Hugging Face’s and ServiceNow’s StarCoder.