Configuring and Testing a Prompt Agent
You’ve created an agent in the Microsoft Foundry portal for an internal support team. Its job sounds manageable. Given a staff member’s question, it should provide a concise answer based on the organization’s support process.
The agent saves successfully, and its first response looks polished. Then you test a similar question and get a different kind of answer. One response is brief and cautious.
Another invents details that weren’t provided. A third ignores the requested format. Is the agent ready because the configuration saved, or do those results tell you something more important?
Creating an agent and validating an agent are separate parts of the work. A saved configuration proves that the platform accepted that configuration. It doesn’t prove that the selected model, instructions, and any connected tools will produce the intended behavior across the workload you care about. To make that judgment, you need to test representative inputs, observe the outputs, and compare them with a clear expectation.
AI-901 includes a single-agent solution in the portal. Microsoft Foundry Agent Service is a managed platform for building, deploying, and scaling AI agents. Managed doesn’t mean that the platform decides what your agent should accomplish. You still need to define the task, choose an appropriate implementation path, configure the agent, and determine how you’ll recognize acceptable behavior.
For a prompt agent, three central configuration choices are its instructions, its model, and its tools. The instructions describe the role, task, constraints, and expected response behavior. The model supplies the generative capability used to interpret input and produce a response. Tools, when needed, let the agent use capabilities beyond generating an answer from the immediate conversation. These parts work together, so a weakness in one can appear as a problem somewhere else.
Suppose your support agent is instructed only to help employees with account problems. That sounds reasonable, but it leaves major questions unanswered. Should it provide a short explanation or a detailed procedure? What should it do when required information is missing?
Should it distinguish established policy from a suggestion? What topics are outside its role? The model can’t reliably infer every intended organizational boundary from a broad sentence. Clearer agent instructions give it a more precise operating context.
That doesn’t mean longer instructions are automatically better. Length isn’t the goal. Useful instructions make the required behavior easier to distinguish from unacceptable behavior. For this support task, you might define the audience, limit the agent to the supplied support context, require it to acknowledge missing information, and specify the form of the answer. You’re turning a vague intention into conditions that can be tested.
The implementation path matters too. Foundry supports prompt agents that are primarily defined through instructions, models, and tools. It also supports hosted agents for situations that require custom code and greater control over the runtime.
If the requirement is to configure and evaluate one straightforward agent through the portal, a prompt agent may fit the task. Choosing a hosted agent merely because it offers more control could add complexity without addressing the actual problem. On the other hand, a requirement for custom runtime behavior shouldn’t be forced into a prompt-only design just because the portal path looks simpler.
The distinction comes from the workload, not from which option sounds more advanced. Ask what input the agent receives, what output it must produce, what capability it needs, and what constraints govern the result. If instructions, a suitable model, and supported tools can express the needed behavior, the prompt-agent path aligns with that requirement. If the requirement depends on custom code and runtime control, the hosted-agent path addresses a different need.
Now return to the internal support agent. Its input is a natural-language question from an employee. Its desired output is a concise, bounded response that follows the organization’s stated support process.
The first design decision is therefore not which tool to add. It’s whether the agent can perform the basic language task under clear instructions with an appropriate model. Adding tools before identifying a tool-dependent requirement makes the design harder to interpret and gives you more possible causes when testing fails.
Before testing, define what success would look like. For a question containing all required details, perhaps the answer should identify the relevant support action without inventing facts. For an incomplete question, the agent should acknowledge the missing information instead of pretending certainty.
For a request beyond its assigned role, it should stay within the boundary established by its instructions. These expectations form an input and output contract. They give you something more useful than a general impression that the answer sounds good.
You can then use the portal’s testing capability to submit representative prompts and observe what the agent does. Start with a normal request that clearly fits the intended workload. If the output follows the expected content, boundary, and format, that’s evidence for this particular case.
It isn’t proof of universal correctness. Next, test a reasonable variation in wording. A useful agent shouldn’t depend on one memorized sentence structure to recognize the same underlying request.
Then test an incomplete request. Imagine an employee says, “My account doesn’t work. Fix it.”
The agent may not have enough information to identify the applicable support action. If your instructions say to acknowledge missing information, the expected response should reflect that boundary. An answer that confidently assumes a specific cause may sound helpful while violating the requirement. This is why plausibility and correctness aren’t the same thing.
Finally, test a nearby request that should be handled differently. If the agent is limited to explaining the support process, ask it to perform something outside that role. The exact boundary depends on the hypothetical instructions you established, but the purpose of the test is consistent. You’re checking whether the agent follows the defined constraint when the user’s request pulls it in another direction.
Suppose those tests produce inconsistent results. The first prompt gets a concise answer. The reworded prompt produces unnecessary detail.
The incomplete prompt leads to an unsupported assumption. Don’t respond by changing everything at once. If you replace the model, rewrite all the instructions, and add a tool in one attempt, a later improvement won’t tell you which change mattered. Controlled refinement creates clearer evidence.
Begin with the mismatch between expected and observed behavior. The incomplete prompt exposed an uncertainty problem, so refine the instruction governing missing information. Make the expected response more explicit.
The formatting variation exposed a presentation problem, so clarify the requested response form. Retest the original prompts after each meaningful revision. This lets you see whether the targeted behavior improves and whether a change damages behavior that was already acceptable.
Model choice is another part of the configuration, but it shouldn’t become a substitute for examining the requirement. If behavior conflicts with the instructions, consider whether the instructions are ambiguous, whether the selected model supports the task adequately, and whether the expected result is realistic for the chosen implementation. The right diagnosis comes from evidence. Automatically switching models whenever an answer disappoints you can hide an unclear task definition rather than correct it.
Tools should receive the same disciplined treatment. A tool belongs in the design when the agent requires that capability to satisfy the workload. Its presence doesn’t guarantee correct use, and a successfully saved tool configuration doesn’t prove that the resulting behavior matches your intention. Testing must exercise the condition in which the tool is relevant, then compare the observed result with what the task requires. If the scenario doesn’t require a tool, leaving it out keeps the agent simpler and reduces the number of moving parts under evaluation.
Foundry Agent Service also includes enterprise capabilities involving identity, networking, observability, and safety controls. Those capabilities matter around an agent solution, but they don’t eliminate the need to test the agent’s task behavior. An identity control answers a different question from whether the instructions produce an appropriately bounded response. Observability can help you examine behavior, but observing an output isn’t the same as declaring it correct. Safety controls contribute to the overall solution, while the specific workload still needs its own expected results and validation.
This separates adjacent controls that are easy to blur together. Configuration establishes what you’ve asked the service to use. Identity and networking help govern access and connectivity.
Safety controls contribute protective boundaries. Observability provides evidence about operation and behavior. Agent testing compares the behavior you receive with the behavior the workload requires. These concerns can support one another, but none should be treated as proof of all the others.
The most dangerous result may not be an obvious failure. An error is visible and demands attention. A fluent answer can pass unnoticed even when it doesn’t fully satisfy the requirement.
Consider the support agent responding to an incomplete request with a detailed procedure based on an assumed cause. The response is readable, relevant to account support, and confidently presented. Yet it failed the input and output contract because the necessary information wasn’t available and the agent didn’t acknowledge that limitation.
This exposes the misconception that saving an agent guarantees its instructions and tools will behave as intended. Saving tells you the configuration was accepted. Intended behavior is a broader claim that requires observed evidence.
The lowest-complexity test is often a small set of carefully selected prompts that distinguish the two interpretations. Include a normal case, a reworded case, an incomplete case, and a boundary case. You don’t need a complicated evaluation merely to discover that the agent ignores a central instruction.
When a test fails, name the assumption before making a correction. Perhaps you assumed the phrase “help with account problems” was specific enough. The output shows that it wasn’t sufficient to control uncertainty or format. Revise the relevant instruction, keep unrelated settings stable, and run the same tests again. If the incomplete request now receives an appropriately limited response while the normal request still works, you’ve gained evidence that the revision addressed the observed problem.
Be careful about what that evidence does and doesn’t establish. A few successful tests support the conclusion that those cases behaved as expected under the tested configuration. They don’t establish that every future wording, tool interaction, or operating condition will produce the same result. Record remaining uncertainty rather than turning a narrow test into a universal guarantee. This is especially important when an output is probabilistic and when the workload can contain varied natural-language input.
A useful test set should represent the intended workload instead of being designed only to make the agent look successful. If every prompt closely repeats the wording in the instructions, you may learn very little about how the agent handles realistic variation. Representative prompts keep the underlying requirement stable while varying the way a person might express it. Boundary prompts test whether the agent remains within its assigned role. Together, they reveal more than repeatedly submitting one ideal request.
The same reasoning helps when you need to choose between refining the current agent and changing the implementation approach. If the problem is an ambiguous response rule, improving the instructions is a direct correction. If the task requires a capability the current configuration doesn’t provide, instruction editing alone won’t create that capability. If the requirement genuinely calls for custom code and greater runtime control, a hosted agent may be the appropriate path. The observed failure must be connected to the actual missing requirement before you select the remedy.
That decision chain is the required outcome, the matching capability, the implementation choice, and the observed result. First state what the solution must receive and produce. Then identify the capability that can perform that work under the relevant constraints.
Choose the simplest implementation path that fits those needs. Finally, verify the observed behavior against a result you defined before testing. Product-name recognition alone can’t complete that chain because it doesn’t explain why a capability fits or how success will be demonstrated.
Responsible implementation also means considering the boundaries around the workload, rather than evaluating only answer style. Ask what the agent should and shouldn’t do, what information is appropriate for the task, and which controls are relevant to the solution. Within this episode’s scope, the key point is that those constraints become part of your selection and validation criteria. A response can be well formatted and still fail because it crosses the role boundary you intended to enforce.
For an AI-901 scenario, you should be able to explain the whole decision in connected terms. The clue is that one agent must answer internal support questions under defined instructions. A prompt agent fits when the needed behavior can be expressed through its model, instructions, and any genuinely required tools. A hosted agent would be the adjacent alternative when custom code and greater runtime control are required, not merely when the first prompt-agent test is imperfect. The evidence comes from representative portal tests whose outputs are compared with the expected task, format, and boundary.
If a scenario says only that the agent was created successfully, you don’t yet have behavioral verification. If it says the agent produced one acceptable response, you have evidence for that response, but not a guarantee covering the entire workload. If it says representative normal, incomplete, varied, and boundary inputs were tested against defined expectations, the conclusion is stronger because the implementation has been examined in the conditions it’s meant to handle.
The portal makes it practical to create the configuration and observe responses without beginning with a custom runtime. That’s valuable for foundational implementation, but the interface isn’t making the architectural judgment for you. You decide what the agent is responsible for, why its implementation path fits, what evidence counts, and which uncertainty remains after testing. The managed service supplies capabilities. Your requirement and validation process give those capabilities a defensible purpose.
So the support agent isn’t ready merely when it saves, or even when its first answer sounds convincing. It becomes a credible implementation when its instructions express the intended behavior, its model and tools fit the required capability, and its responses hold up against representative tests. When the evidence exposes a mismatch, refine the relevant configuration and test again. That cycle turns an agent from a saved collection of settings into a solution whose behavior you can explain and evaluate.
Creating and testing a single agent comes down to a disciplined connection between intent and evidence. Start with the required input, output, capability, and constraints. Choose a prompt agent when instructions, a model, and supported tools fit that requirement, rather than selecting a more complex path without a reason. Then test normal, varied, incomplete, and boundary requests against expectations you defined in advance.
A saved configuration shows that Foundry accepted the setup. Only observed behavior can show whether the agent is following it in the cases you tested. When the result falls short, change the part connected to the failure and retest. That’s how you replace a plausible first response with a defensible single-agent implementation.
Before moving on, ask yourself: Which test would show that a saved agent configuration still fails the intended support job? Use the companion episode reference for portal creation, model and instructions, supported tools, representative tests, and boundary behavior. This lesson was produced by Bare Metal Cyber dot com.