Kyotango City, Kyoto Prefecture
Area: 501.84 km²
Population: 51,095 (as of November 30, Reiwa 5)
Kyotango City is located on the Tango Peninsula in the northern part of Kyoto Prefecture, spanning approximately 35 km east to west and 30 km north to south. It is a region blessed with the sea, mountains, and countryside. The coastal area is designated as the Tango-Amanohashidate-Oeyama Quasi-National Park and the San'in Kaigan National Park and is a member of the Global Geoparks Network as the San'in Kaigan Geopark. The region is also rich in natural hot springs, which are the oldest in the prefecture and have the highest number of hot spring sources. Additionally, the region is known for its abundant agricultural and marine products, such as "Tango Rice," "Taiza Crab," "Kyoto Vegetables," and "Kyotango Pears."
In a survey, 78% of the city employees reported increased work efficiency using Crew, and 64% of the employees expressed that the introduction of generative AI is necessary for the city's future.
We spoke with the members of the Kyotango City Young Employees Project about the background, benefits, and feedback on the demonstration experiment of “Crew”, a safe use of ChatGPT within the office.
Please tell us about the background of conducting the Crew trial.
In Kyotango City, we have formulated a DX strategy to enhance citizen services and promote digitalization within the administration. To improve operational efficiency, we have been actively utilizing advanced technologies such as RPA and AI-OCR. Additionally, to harness the flexible thinking and innovative ideas of young employees for city governance and community development, we have been implementing the "Young Employees Policy Proposal Project" since FY2021. The young employees' team has also been considering ways to utilize generative AI.
In this context, we believed that trying out the service and using the insights for further discussions was the best way to determine how to effectively utilize generative AI for specific tasks. Therefore, we decided to conduct a trial. The trial was timely, coinciding with a press conference where we also demonstrated Crew. The journalists’ response was generally positive.
Why did you choose Crew among the various ChatGPT services available to municipalities?
We were particularly interested in Crew's ability to generate internal organizational documents, a feature not widely offered by existing services, which led to our selection. During our first meeting, we were shown a demo screen that gave us a clear idea of how to use it, such as generating responses from internal Q&A databases. c. In addition, the ability to check usage logs through the admin panel and on-screen warnings for entering personal information was also appealing.
Furthermore, although this is not directly related to why we chose Crew, after expressing the request for having a user community, an exchange meeting with other municipalities using Crew was organized. This has provided us with opportunities to learn about the current state and challenges of generative AI. There are not many opportunities to learn about use cases for generative AI in other municipalities. Therefore, expanding connections through Crew and sharing information, such as examples of prompt usage, in document channels within the community would be extremely helpful.
Reference: First Crew User Meeting Held!
https://www.gocrew.jp/news/crew-user1
We will continue to develop the user community! How did you use Crew?
In this trial, we aimed to evaluate the convenience of generative tools by responding to internal documents. Therefore, we used Crew for tasks such as creating, summarizing, and proofreading of internal documents, drafting public relations documents for residents, and creating FAQs based on internal manuals. To broadly verify its usefulness within the office, we provided accounts to 30 departments. At the beginning of the trial, we held hands-on experience sessions to explore the different tasks that Crew could be utilized for.
Thirty departments! That’s the highest number of departments that have participated in a Crew trial to date (laughs). What kind of documents did you upload and use?
For example, in a channel where we uploaded the minutes of our city council meetings, we asked, "Please list five key points from each council member's questions." The response was as follows: "Council Member XX's key points: 1. Considering listening to enthusiasts' opinions during research. 2. No budget allocated for Fiscal Year 5, planning internal research and studies. 3. ..., Council Member YY's key points: 1. Proceeding with interviews with enthusiasts and considering internally...". In this way, Crew summarized the key points of each speaker during the council meetings.
Before drafting responses to council inquiries, there is a need to summarize and organize past discussions on specific issues in every department. It suggests that Crew could be widely deployed within our office. Depending on the content and volume, summarizing and organizing key points from council minutes, which used to take about an hour to summarize and organize, could be reduced by half with Crew. Additionally, the minutes were recorded in spoken language, but Crew's ability to convert them into coherent written form was pleasantly surprising.
It seems that Crew can indeed contribute to the efficiency of tasks unique to municipalities. How else have you utilized it?
We also tested Crew by uploading our city's electronic approval operation manuals to verify its ability to generate accurate responses. For example, when it comes to a manual in tabular format, we asked the following questions from the section, "What is the answer to the following question from the section 'Opinions from each department regarding implementation’: What is the response from the General Affairs Division regarding operational improvements during circulation?" The correct response we received was: "The General Affairs Division’s response regarding operational improvements during circulation is that it seems to be an issue of becoming familiar with the operation, so we will add this to the manual. Also, ...'" Even when presented in a table format, as long as the information is organized into coherent sentences, the comprehension accuracy was quite good. This suggests that we could consider deploying it for information query tasks within the office.
However, the reading accuracy for manuals with many inserted screenshots was not as high, so we hope for improvements in this area in the future.
Other than the document channels, how did you try Crew in text channels?
In text channels, for instance, when brainstorming policy proposal titles, we come up with them from scratch. With Crew, we generated several title suggestions and combined them to finalize the title. This allowed us to integrate it into our existing workflow. It seemed like we reduced the workload by about one-third compared to coming up with ideas manually from scratch.
Additionally, we enlisted the help of Crew in creating external PR articles. By asking Crew to help with phrasing and the flow of the text, we managed to reduce the time required to complete an article from the usual 2-3 hours to about 30 minutes.
What was the reaction of the staff who actually tried Crew?
We conducted a survey among the staff who used Crew, asking, "Do you think Crew has contributed to improving work efficiency?" As a result, 78% of the staff responded that they think "work efficiency has improved." Interestingly, more than 60% of the staff who participated in this trial were using generative AI services for the first time. Despite this, approximately 80% of the members recognized that it could enhance their work efficiency, which is promising. Additionally, when asked, "Do you think the introduction of generative AI like ChatGPT is necessary for Kyotango City in the future?" 64% of the staff responded "necessary." Of course, the necessity varies depending on the nature of the work, but through Crew's trial, many staff members were able to recognize the usefulness of generative AI. Based on these trial results, we would like to consider implementing generative AI services for external uses, aiming to improve citizen services, in addition to internal uses.
Thank you. We have heard positive examples, but could you also share any instances where things didn't go well?
Overall, while trying out Crew was a good experience, the challenge was how to ensure its continuous use afterwards. We conducted hands-on experience sessions within the office to foster a "let's give it a try" atmosphere. However, we faced difficulties in individual follow-ups, which led some staff members to stop using the system after a few attempts when the expected answers were not generated. I believe the difference in expectations regarding generative AI was a significant factor. Therefore, aligning expectations and creating a mechanism to support those who have discontinued use will be crucial moving forward. It would be great if we could continue exchanging information in the Crew user community regarding these aspects.
Additionally, when multiple documents are uploaded to a single channel, the accuracy of the responses tends to decrease. Therefore, it is also necessary to consider how many documents should be uploaded to each channel.
Do you have any suggestions for future improvements?
To familiarize users with Crew’s features, I believe it’s crucial to become accustomed to prompts. In this regard, it would be beneficial if the prompt templates provided by Crew were better suited to our daily tasks. Since there is a wide range of literacy among staff, having prompts that are easy to use, even for those unfamiliar with the system, would likely increase the frequency of usage. I understand that there are plans to implement a feature allowing administrators to freely add templates in the future, and I am eager to try it out.
Additionally, while it is indeed convenient to generate responses from documents, I believe that if responses can be generated by combining document data with ChatGPT's training data, it would greatly expand the scope of applications. Currently, using only document data limits the utility of tasks such as confirming internal manuals and regulations, which can sometimes feel insufficient to users. By mixing our city’s data with ChatGPT's data in the future, such as referencing city guidelines while examining differences with other municipalities, I think the overall convenience would significantly improve.
Considering your feedback, we aim to further improve. Thank you very much for your time!