-p-800.png)
.png)
About Kashiwazaki City, Niigata Prefecture
Area: 442.03 km²
Population: 77,753 (as of September 30, Reiwa 5 [2023])
Kashiwazaki City, located nearly at the center of Niigata Prefecture and facing the Sea of Japan, is a regional city with a population of just under 80,000. It is home to the sacred Mt. Yoneyama, which is famous for the folk song "Sankai Bushi ," and the Kariwa Sanzan, consisting of Mt. Kurohime and Mt. Hachikoku . The city boasts a 42 km coastline that includes the Fukura Eight Scenic Views and sand dunes, along with 15 beaches. Kashiwazaki is a town nurtured by rich nature, scenic beauty, traditional culture, and history, featuring places like Ukawa, where the national important intangible folk cultural property "Ayako Mai" is performed; Takayanagi, known for its terraced rice fields and thatched-roof circular settlements; and Nishiyama, recorded in the Nihon Shoki for offering "burning water" to the Imperial Court.
For this case study, we spoke with Mr. Hiroyasu Abe, Section chief of the Information and Statistics Section, General Planning Department, Planning and Policy Division in Kashiwazaki City, and Mr. Ryohei Toyama, a Principal Officer, about their experience using "Crew" for safely implementing ChatGPT within the office. They shared insights into the background, effects, and their impressions of the pilot test of "Crew."
Question: I appreciate you taking the time to speak with me. Could you please provide some background information on conducting the pilot study of using Crew?
Answer: Around the spring of 2023, there was increasing media coverage about various local governments across the country starting to utilize ChatGPT for their operations. Our city was also considering the introduction of generative AI services like ChatGPT to streamline municipal tasks. We aimed to optimize internal administrative work and reallocate the time saved to providing services that only humans can deliver. While exploring effective ways to utilize these services, we recognized the need to concretely evaluate the risks and feasibility of such services. Based on this context, our city decided to conduct a pilot study on the use of generative AI in our municipal operations.
Question: Among the various ChatGPT services available to municipalities, can you explain why you opted for Crew?
Answer: While we were gathering information on various services, we chose Crew because it offers services that generate AI content using documents internal to an organization, which are limited in other existing services. . When considering the efficiency of our internal administrative work, it's crucial to utilize AI generation based on our city's unique documents. Therefore, we wanted to test how accurately Crew can generate responses using the documents we have. Another attractive feature of Crew is its comprehensive functions, such as warnings about personal information, which don't solely rely on staff intervention. Even if we create guidelines and restrict the input of personal information, the final operation of the tool depends on the staff. The ability to control this aspect seemed convenient, so we decided to conduct a pilot experiment with Crew.
Question: How have you leveraged Crew's capabilities?Answer: In our meetings with the representatives of Crew, we were particularly drawn to its ability to read internal documents and generate responses. Therefore, this time we focused on creating new documents and generating Q&As based on ordinances and regulations, using our internal documents as a basis. We conducted evaluations and verifications with this focus.
Question: Thank you. Could you specify what kind of documents you uploaded for use?
Answer: For instance, we uploaded basic policy documents related to the standardization of systems issued by the government and requested Crew to create drafts for external explanations. Local governments often deal with voluminous documents and frequently need to distill important points to explain to the public. Therefore, to simulate actual work scenarios, we tested how effectively and accurately Crew could generate documents that concisely capture the essential points.
Question: Could you specify the types of questions that were posed?
Answer: Utilizing the prompt templates implemented in Crew, I asked to summarize the 'reasons for municipalities to advance standardization' in about 100 characters, based on [the following constraints #constraint conditions...]. As a result, Crew generated a response that effectively picked up the key points: 'The reason local public organizations are moving towards standardization is as follows. Compliance with standards during system updates from fiscal 2023 onwards is required.
Additionally, it aims to reduce operational expenses of standardized administrative tasks by 30%' Crew created an output that, with minor adjustments by staff, can be sufficiently used as an external explanatory document. Crew is equipped with a template function to complement prompts, automatically setting specific questions and constraint conditions to elicit more concrete answers. Its design is appealing as it enables even staff unfamiliar with prompts to extract high-quality responses.
Question: What has been the impact since introducing Crew?
Answer: Regarding the documentation related to system standardization mentioned earlier, it would probably take about an hour to create such documents manually from scratch. However, with Crew, it organizes the key points and generates responses, so even considering the time needed for staff revisions, we've managed to reduce the effort to about 30 minutes.
Furthermore, tasks like explaining the contents of these documents to the citizens on the city's website occur at least about 40 times a month. If Crew could automate all these tasks, we anticipate saving about 20 hours per month, which adds up to approximately 240 hours annually.
Question: 240 hours a year! That seems like a considerable improvement in work efficiency. Are there other areas where time has been saved?
Answer: Similarly, in documentation related to system standardization, we referred to Crew for guidance on how municipalities can advance standardization. The documents contain a vast number of procedures, and it takes a significant amount of time for a person to determine which elements are particularly important one by one.
When we tried generating responses through Crew, the results were generally appropriate. If staff was to read and interpret the documentation from scratch and then create a document for other staff members, it would likely take about 3 hours. However, with Crew generating the text and staff adding details as needed, the work was completed much more efficiently. What would have taken 3 hours was reduced to about 1 hour.
Question: Thank you. What kind of question did you ask Crew at that time?
Answer: We also utilized the prompt template feature and asked, 'Based on the following constraints, please list and explain what should be implemented by the fiscal year 2027 for standardization. #Constraint conditions…'. As a response, Crew organized the procedure into a total of five steps, starting with 'Here is a list of things that need to be implemented by the fiscal year 2027. Addressing revisions in the specification document...' Additionally, Crew has a feature that highlights which part of which document was referenced in producing the answer, which was extremely convenient. Even if AI generates the text, final verification by humans is necessary, so I found this feature essential for scrutinizing the consistency of the generated information."
Question: In what other innovative ways could this tool be applied or utilized??
Answer: We used Crew to confirm the regulations concerning travel expenses for our municipal employees by feeding it the manual related to the operation of travel expenses. Although the original document was organized in a Q&A format, finding straightforward answers was challenging since the regulations change depending on various factors such as the place of departure, distance of the business trip, or mode of transportation.
For instance, when we asked Crew a question like, 'If I travel by train from Kashiwazaki City Hall to XX City Hall, will travel expenses be paid?' Crew generated a response considering multiple conditions: 'If you travel by train... If you move within the same area... If you move outside the same area... Since XX City is outside the same area, if you travel by train... will be paid.' Instead of having to open a PDF file each time and search for the relevant section, Crew provides answers considering the conditions, which we felt reduced the time required for confirmation to about a third of what it used to be. Furthermore, as business trips can occur in any department, we anticipate a reduction in work hours at an organizational level."
Question: Thank you. We have heard some good examples so far, but I would also like to know about any instances where it didn’t work well. Could you share any such experiences?
Answer: It was beneficial to have the prompt template feature. However, when we ask abstract questions without using templates, the answers we get back tend to be high in abstraction too, so some ingenuity is needed in this area. Additionally, I felt that there was a variance in response accuracy depending on the file type. The documents regarding standardization and travel expense regulations that we discussed earlier were well organized in text format, but when the data included non-text elements, we didn’t always get the expected responses, leading to situations where there was less prospect of improving operational efficiency. Therefore, it would be great if the tool could uniformly generate answers from any type of file.
Question: Do you have any suggestions for improvement?
Answer: I felt it would be more convenient if users could add their own prompt templates. As each department has different work contents, for instance, it would be beneficial to have the ability to freely add prompts suited to each department's tasks. For example, the Commercial and Tourism Department could have templates for brainstorming taglines, or the Citizens' Affairs Department could have templates for creating surveys. I believe this feature would greatly expand the potential uses of Crew. I've heard that this functionality is planned for future implementation, and I definitely would like to try it out.
Additionally, I am looking forward to a future where we can track the usage of Crew not just at the organizational level but also at the individual user level. This would allow us to identify and focus on those who are actively using Crew, learning from their usage patterns. Conversely, if there are staff members who use it less frequently, we can review their usage and potentially reallocate their access to other staff members, thereby achieving a more cost-effective use of the tool.
Based on the feedback received, we are looking to make further improvements. Thank you very much for your time!