.png)
.png)
Abiko City, Chiba Prefecture
Area: 43.15 km²
Population: 131,255 (as of November 1, Reiwa 5)
Abiko City is located approximately 30 kilometres from central Tokyo, nestled between the Tone River to the north and Lake Tega, the nearest natural lake to the city, to the south. Beloved by historical literary figures, the city is vibrant with various events throughout the year. Despite being just a 35-minute train ride from central Tokyo, Abiko City offers the charm of nature and history.
In a survey, 69% of the city's employees responded that they could use Crew to improve work efficiency. Comments included statements such as, " The search results are clearly and thoroughly explained in the text," and "It can be used to create explanatory documents for citizens who lack specialised knowledge."
We spoke with Kousuke Fujita from Abiko City’s Digital Strategy Office about the background, effects, and his impressions of the Crew trial, which tests the use of ChatGPT within the office.
Please tell us about the background of conducting the Crew trial.
Since 2023, generative AI usage has been advancing nationwide in municipalities, Abiko City wanted to experimentally verify how it could be utilised in various tasks. While generative AI has the potential to significantly contribute to the efficiency of internal operations and improve convenience for citizens, we recognize the need to specifically assess what each service can achieve, given the daily emergence of various generative AI services.
When considering a trial run for actual operational use, we heard from a Crew representative and became interested in their service.
Why did you choose Crew among the various ChatGPT services available to municipalities?
We were attracted to Crew because of its capability to generate responses from documents, which is not commonly offered by existing services. Additionally, Crew has account management features, sensitive information alert functions, and the ability to view chats containing sensitive information, making it a highly capable service.
As we plan to expand the use of such services more broadly within our office, it is important to be able to identify the person who asked the question. Regarding alerts, for instance, when using generative AI to draft emails in response to citizen inquiries, there is a risk of accidentally pasting personal names directly. The on-screen alert that automatically changes such text to another string provides a reassuring level of security.
How did you use Crew?
In this trial, we tested both document channels and text channels. For the document channels, we mainly used Crew to generate Q&As based on ordinances and regulations and to draft new documents based on existing internal documents. For the text channels, we used Crew for code creation and text generation.
Thank you. Let's start with the document channels. What specific documents did you upload and use?
We uploaded a total of 1,078 publicly available files of ordinances and regulations from Abiko City. We used these files for document searches, summaries, and summarizing specifications received from the national government to assist in responding to inquiry emails. The majority of the use cases involved searching for document information.
1,078 files! What kind of questions did you actually ask?
For example, in a channel where the city's collection of ordinances was uploaded, we asked: "What points should be considered when digitizing approvals?" The response was, "When digitizing approvals, the points to consider are..., and it should be provided by the following methods (hereinafter referred to in this article as 'electronic methods')." If we wanted more specific information, we could follow up with a question like, "What are the 'following methods' mentioned?" "The 'following methods' refers to the important items that can be provided by electronic methods. Specifically, these include the following items: ..." Crew would extract detailed information from the documents, making it effective for pinpointing the needed information.
In the future, if Crew is able to generate text based on document content in addition to conducting information searches, the benefits of municipalities adopting Crew will further increase.
Thank you. In addition to the document channels, how did you utilize the text channels?
We tested the generation of VBA code for Excel using the text channels. Since the administrative staff frequently use Excel and Word for their tasks, we have been exploring the use of VBA to improve efficiency within the office. As part of this initiative, we asked Crew to create VBA code. For example, we requested, "Please create a VBA code to consolidate multiple Excel files into one." We ran the generated code and, if there were any errors, we asked Crew to fix them by saying, "I found an error in the part where X is done, please correct it." We iteratively refined the code with Crew's help until the VBA code functioned correctly. Depending on the task, if we write and modify the code from scratch, it would take about 2 hours to complete a task. However, by using Crew, we can complete the same task three times faster.
You managed to speed up your work three times faster. How else did you test it?
We also used the text channels to create documents related to the national financial situation. Specifically, we generated potential Q&A scenarios for reviewing the settlement of the previous fiscal year. For instance, we asked the question, "Why did the local consumption tax grant for Reiwa 1 decrease compared to the previous year?" Crew responded, "Regarding the decrease in the local consumption tax grant for Reiwa 1 compared to the previous year, the following possibilities can be cited: 1. ..., 2. ..." The response was as accurate as ChatGPT. Using the generated text as a reference, our staff added more details to prepare the official response. This proved to be very useful for our actual operations.
What was the reaction of the staff who actually tried Crew?
We surveyed the staff who used Crew to verify the accuracy of the responses. As a result, 68% of the staff answered that "the responses had been appropriate." While humans must make the final confirmation, having Crew generate preliminary answers in seconds was found to be very convenient compared to researching information from scratch. Furthermore, when asked, "How much does Crew contribute to improving work efficiency?" 69% of the staff responded that "work efficiency has increased." Comments included, "The search results are clearly and thoroughly explained in the text," and "It can be used to create explanatory documents for citizens who lack specialized knowledge." This indicates that Crew can be widely utilized within the office for information searches and drafting of external notification documents.
.png)
On a related note, some staff mentioned in the survey that after thanking Crew for its response, it replied with a joke reflecting the preceding conversation, which made them chuckle. While this is a side benefit, this highlighted the potential for Crew to become a "trusted partner" rather than just a "tool," especially for departments with small teams. As a city, we hope that Crew's response accuracy will continue to improve and that it will become a reliable partner for easy consultation on various matters.
So far, we have focused on positive examples, but could you also share any instances where things didn't go as planned?
In the document channel, there were cases where the expected accuracy of the summary results was not met. For example, when we asked Crew to draft a speech for the mayor to deliver at a speech contest, and then requested, "Please make the greeting a bit softer," the revised text sometimes did not reflect the previous version and created a completely different draft. While there is room for improvement in our prompts, enhancing this aspect of accuracy would make it more convenient.
Additionally, when uploading documents, the accuracy of responses remained high as long as only two documents from the same field were uploaded to a channel. However, when documents from somewhat different genres were uploaded to the same channel, the accuracy of the responses declined. Therefore, it's necessary to consider the scope of documents uploaded to each channel.
Thank you. In addition to response accuracy, do you have any other improvement suggestions for the future?
Although it might be challenging, it would be highly beneficial for our city and municipalities nationwide if Crew could upload city budgets and financial statements and generate explanatory texts based on their content. In Japanese cities, there is a council meeting every quarter. In our city, staff explain the contents of the financial statements at the beginning of the city council meeting. This process involves reading the printed financial statements and creating explanatory texts, which can take more than a day. Additionally, about 30 staff members provide explanations at each city council meeting, resulting in a total of approximately 30 man-days. Since this happens four times a year for regular council meetings, we spend about 120 man-days annually preparing these explanatory texts.
If Crew could take over this task and generate draft explanatory texts for each budget item, such as "For general administration expenses..., for social welfare expenses...," it would be incredibly helpful. Even if the work time is halved, it would save about 60 man-days annually, offering significant cost-effectiveness.
Taking your feedback into consideration, we aim to continue making further improvements. Thank you very much for your time!