Add com.codename1.ai package, ChatView, Speech/TTS, and build-time AI dependency injection#5035
Open
shai-almog wants to merge 33 commits into
Open
Add com.codename1.ai package, ChatView, Speech/TTS, and build-time AI dependency injection#5035shai-almog wants to merge 33 commits into
shai-almog wants to merge 33 commits into
Conversation
… dependency injection Introduces the full AI/LLM surface for Codename One plus the build-server plumbing that auto-wires every native dependency it needs. Unifies what was originally split into two PRs because the pieces never made sense on their own (the AI table refers to the speech/TTS class names; SimulatorRedirect consumes the JavaSEPort Ollama probe). ### com.codename1.ai - LlmClient with static factories for OpenAI, Anthropic, Gemini, Ollama, and a generic OpenAI-compatible endpoint. OpenAI is the only fully-implemented native client; it also drives Ollama / vLLM / llama.cpp because the wire format matches. Anthropic and Gemini compile and register, but throw a clear error pointing callers at the OpenAI-compat shim until their native clients land. - Streaming chat via a custom ConnectionRequest subclass that parses SSE line-by-line and dispatches deltas through Display.callSerially. AsyncResource.cancel() kills the socket. - Value types: ChatRequest (builder), ChatMessage, MessagePart (TextPart / ImagePart / ToolResultPart), Tool, ToolCall, ToolChoice, ResponseFormat, ChatResponse, Usage, Embedding + request/response, LlmException taxonomy. - ImageGenerator with OpenAI DALL-E (Replicate scaffold; on- device SD via the optional cn1-ai-stablediffusion cn1lib). - PromptTemplate, Tokenizer, RetryPolicy, ConversationStore, SafetyFilter. ### com.codename1.media.SpeechRecognizer + TextToSpeech - New core APIs routed through Display + CodenameOneImplementation no-op hooks. Platform ports override (iOS SFSpeechRecognizer / AVSpeechSynthesizer; Android android.speech.*). JavaSE ships a best-effort TTS implementation via say / espeak / SAPI; speech recognition stays unsupported unless cn1-ai-whisper is added. ### SecureStorage non-prompting overloads - Single-argument get/set/remove(account, ...) added next to the existing biometric-gated methods. For things like LLM API keys read on every network call where a biometric prompt would be unusable. Base class returns null/false on unsupported ports. ### com.codename1.components.ChatView - Scrollable message list (BoxLayout.Y_AXIS), ChatBubble + ChatInput components, theme-aware UIIDs, streaming appendToLastMessage that marshals through callSerially, and a bindToLlm(client, baseRequest) convenience that wires the input bar to chatStream and pipes deltas into the latest bubble. ### Build-time scanner additions - AiDependencyTable in the maven plugin: 18 entries mapping the com.codename1.ai.* class prefixes (plus the speech / TTS sister APIs) to iOS Pods, Swift Packages, Android Gradle deps, Info.plist usage strings, and Android permissions. - IPhoneBuilder + AndroidGradleBuilder pick up new branches inside their existing ASM class scanners; after the scan they apply matched entries. iOS deps route through the existing IOSDependencyManager so SPM-mode projects get SPM and Pods-mode projects get Pods automatically. Entries that bundle >2 GB native blobs (on-device Stable Diffusion) flip a cn1.ai.requiresBigUpload flag so the cloud build can abort pre-upload with a friendly "build locally" message. ### JavaSE simulator: Ollama detection - Probe at startup pings localhost:11434; sets cn1.ai.ollamaDetected when reachable. SimulatorRedirect reads the property and, with cn1.ai.simulatorRedirect=auto or =ollama, routes any LlmClient.openai(...) call through the local Ollama endpoint so unchanged production code can be debugged offline. ### Tests - 9 plugin tests (AiDependencyTableTest): pods, SPM routing, big-upload flagging, accumulator dedup, false-positive guard. - 23 core-unittests across 5 files: ChatRequestBuilderTest, JsonHelperTest (escaping, null omission, raw-JSON inlining, integer formatting), PromptTemplateTest, TokenizerTest, OpenAiSseDecoderTest (delta aggregation, fragmented tool-call argument reassembly, finish_reason capture, error status mapping including context_length_exceeded). ### scripts/create-ai-cn1lib.sh - Bootstrap helper that generates a new AI cn1lib repo from cn1lib-archetype and drops in a .github/workflows/publish.yml that publishes to Maven Central on every merge to master. ### Tracked follow-ups - iOS / Android native bridges for SpeechRecognizer + TextToSpeech. - BuildDaemon mirror of the scanner additions (separate PR). - Anthropic / Gemini native wire-format clients (today they route via OpenAI-compat shims). - The thirteen cn1-ai-* cn1lib repos themselves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 24, 2026
Collaborator
Author
|
Compared 11 screenshots: 11 matched. |
Collaborator
Author
JavaScript port screenshot updatesCompared 22 screenshots: 20 matched, 2 missing references.
|
Collaborator
Author
Android screenshot updatesCompared 112 screenshots: 110 matched, 2 missing references.
Native Android coverage
Benchmark ResultsDetailed Performance Metrics
|
The static-analysis gate (.github/scripts/generate-quality-report.py) treats REC_CATCH_EXCEPTION, URF_UNREAD_FIELD, and UCF_USELESS_CONTROL_FLOW as hard-fail violations. Cleaned up the AI code accordingly: - OpenAiClient.postResponse + embed.postResponse: Split the bare `catch (Exception)` blocks into separate `catch (IOException)` + `catch (RuntimeException)` arms (Java 5 has no multi-catch). The IOException path covers JSON parse failures from JsonHelper.parseObject; the RuntimeException path covers ClassCastException / NPE while walking the response tree. Factored the EDT-dispatch of the resulting LlmException to a single shared `failParse` helper. - OpenAiClient.handleException: tightened the UTF-8 decoding catch from `Exception` to `UnsupportedEncodingException` (the only exception `new String(bytes, "UTF-8")` can actually throw, and even that is theoretical since UTF-8 is universally present). - OpenAiSseDecoder.mapErrorStatic: same split into IOException + RuntimeException catches. - ImageGenerator.postResponse + handleException: same split treatment. - ImageGenerator.ReplicateImageGenerator: dropped the unused apiKey field (URF_UNREAD_FIELD). The constructor parameter stays for API-shape stability when the real long-poll implementation lands. - ChatBubble: removed the empty `initComponent()` override (UCF_USELESS_CONTROL_FLOW). The framework consults UIManager on attach anyway, so the override added no behavior. Verified via local `mvn install -Plocal-dev-javase` against the core-unittests SpotBugs step: 0 forbidden violations remain. The 2 leftover findings (RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE and WMI_WRONG_MAP_ITERATOR) are style-level and not in the gate's forbidden set. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
iOS screenshot updatesCompared 112 screenshots: 110 matched, 2 missing references.
Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
Collaborator
Author
iOS Metal screenshot updatesCompared 112 screenshots: 110 matched, 2 missing references.
Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
The CI static-analysis gate also enforces RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (line 841 of .github/scripts/generate-quality-report.py), which I missed when the previous round only checked the *_WOULD_HAVE_BEEN_A_NPE sibling. Util.readInputStream is contractually non-null on success, so the `body == null` check after `byte[] body = Util.readInputStream(input)` is dead code. Removed it. Verified locally: 0 forbidden bug instances remain in the spotbugsXml.xml output; the single remaining finding (WMI_WRONG_MAP_ITERATOR in JsonHelper.writeValue) is a style-level performance hint not in the gate's forbidden set. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Build-test (8) runs PMD via maven-pmd-plugin in core-unittests and feeds every violation through the same forbidden-rule gate the SpotBugs path uses. Cleaned up 87 violations across nine rule categories. Verified locally with mvn verify in core-unittests: zero violations. - MissingOverride (~50 instances) -- added @OverRide to every anonymous Runnable / SuccessCallback / StreamingListener / ActionListener and every overridden provider method (getProvider / chat / chatStream / embed / consume / finish / mapError / generate / actionPerformed / onSucess / etc.) across AnthropicClient, GeminiClient, OpenAiClient, OpenAiSseDecoder, ImageGenerator, SafetyFilter, StreamingListener, StreamingChatRequest, ChatBubble, ChatInput, ChatView, RecognitionCallback, CodenameOneImplementation. - ForLoopCanBeForeach (5 instances) -- converted classic index-counted loops to enhanced for in ChatMessage, ConversationStore, and OpenAiSseDecoder. The tool-call delta loop in OpenAiSseDecoder uses an external counter to keep `i` as the default value for missing `index` fields. - ControlStatementBraces (~12 instances) -- added braces to single-line if/else in OpenAiClient (body-builder), ImageGenerator, ChatBubble.defaultUiidFor, and ConversationStore.parseRole. - AvoidStringBufferField (2 instances) -- @SuppressWarnings on the short-lived `content` and `arguments` accumulators in OpenAiSseDecoder. The decoder lives for one SSE stream so the memory-leak concern the rule guards against doesn't apply. - EmptyCatchBlock (1) -- added Log.e on the listener.onError swallow in StreamingChatRequest.failWith. - SimplifyBooleanReturns (1) -- collapsed the trailing `if (...) return true; return false;` in RetryPolicy.shouldRetry to a direct `return t instanceof LlmNetworkException`. - UnnecessaryFullyQualifiedName (4) -- replaced inline `java.io.IOException`, `java.io.UnsupportedEncodingException`, `com.codename1.ui.Display.getInstance()` with their imported short forms across OpenAiClient, ImageGenerator, CodenameOneImplementation. - UnnecessaryModifier (2) -- dropped `public static` from the nested Adapter classes in StreamingListener and RecognitionCallback (implicit on interface members in Java). - UnusedFormalParameter (1) -- @SuppressWarnings on ImageGenerator.ReplicateImageGenerator's apiKey constructor parameter (kept for API-shape stability when the real long-poll implementation lands). Also removed an obsolete Arrays.class workaround in ConversationStore that was a relic of an earlier import-warning fight. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build-test (8) also runs Checkstyle (build-test (17) ran PMD but not Checkstyle in the same path -- both gates must pass). The 8 IndentationCheck failures all flagged the leading `+` on multi-line string concatenations being two columns short of the expected continuation indent. Indented the continuation lines by 2 extra spaces in the multi-line UnsupportedOperationException messages in: - AnthropicClient.chat / .embed - GeminiClient.chat / .embed - ImageGenerator.onDevice / .ReplicateImageGenerator.generate Verified locally via `mvn checkstyle:check`: 0 violations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
Acts on six pieces of feedback raised on PR #5035: 1. **Drop scripts/create-ai-cn1lib.sh** -- redundant now that the cn1libs ship in-tree. 2. **Fold JsonHelper into JSONParser**. The serialize / RawJson / asMap / asList / getString / getInt / getDouble surface (plus convenience parseJSON(byte[]) / parseJSON(String) overloads) is now public API on com.codename1.io.JSONParser. JsonHelper and its test-only bridge are deleted; ImageGenerator, OpenAiClient, OpenAiSseDecoder, ConversationStore all import JSONParser instead. 3. **Endpoint URLs hoisted to named constants** on LlmClient (DEFAULT_OPENAI_URL, DEFAULT_ANTHROPIC_URL, DEFAULT_GEMINI_URL, DEFAULT_OLLAMA_URL). Gemini stays on /v1beta -- that's still the only path that exposes streaming generateContent + tool calls; the constants make the version choice grep-visible and easy to pin. setBaseUrl(...) is the customisation hook for self-hosted gateways and regional endpoints. 4. **LlmException.ErrorType enum** so callers can do a single try/catch + switch (the more idiomatic shape for most apps): switch (e.getType()) { case RATE_LIMIT: ... case AUTH: ... case CONTEXT_LENGTH: ... ... } The subclasses (LlmRateLimitException etc.) remain for cases where instanceof + extra typed state (e.g. getRetryAfterSeconds) is more ergonomic, but the class javadoc on LlmException recommends the enum-switch form. Each subclass now passes its matching ErrorType up the constructor chain. 5. **Tool / ToolCall linkage**. Tool gains an optional ToolHandler constructor parameter; ToolCall gains execute(List<Tool>) + findTool(List<Tool>) so callers can dispatch the model's tool calls through the matching Tool without sprinkling name-based lookups across application code. Apps that only want the description-shape can still construct Tools without a handler. 6. **ChatView decoupled from LlmClient**. Moved bindToLlm into a new com.codename1.ai.LlmChatBinding helper class. ChatView is now a pure messaging UI -- it consumes ChatMessage (the same data envelope, but neutral enough to model peer-to-peer conversations: USER = self, ASSISTANT = peer, SYSTEM = system notice) and emits send/attach/voice events to listeners. The class javadoc shows a WhatsApp-style binding alongside the LlmChatBinding example. Plus **13 AI cn1lib scaffolds** under maven/cn1-ai-<feature>/: cn1-ai-mlkit-text / -barcode / -face / -labeling / -translate / -smartreply / -langid / -pose / -segmentation / -docscan, cn1-ai-tflite, cn1-ai-whisper, cn1-ai-stablediffusion Each ships a public Java facade in the corresponding com.codename1.ai.<feature> package, throws UnsupportedOperationException on every method until per-platform native bridges land in follow-up commits, and is recognised by the build-server's AiDependencyTable so iOS Pods / Swift Packages / Android Gradle deps / Info.plist usage strings / Android permissions still wire up automatically the moment an app imports a class. Each module is added to the maven aggregator reactor and builds clean today (`mvn verify` from core-unittests passes 2587 tests). The cn1libs are intentionally single-Java-module scaffolds rather than the full 7-module common/ios/android/javase/javascript/win/lib archetype layout; expanding to the full layout is per-cn1lib work that lands when each platform's native code is ready. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build-test (8) flagged three java.io.IOException / .InputStreamReader / .ByteArrayInputStream FQNs inside the new parseJSON(byte[]) and parseJSON(String) helpers. Added the missing imports and switched to short names. Local mvn verify is clean (2587 tests, 0 PMD). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e pattern
Acts on two follow-up review notes:
1. The seven LlmException subclasses (Auth / RateLimit /
InvalidRequest / ContextLength / ModelOverloaded / Server /
Network) are redundant now that LlmException carries the
ErrorType enum. Deleted all seven; moved retryAfterSeconds (the
only piece of state that lived only on the rate-limit subclass)
onto LlmException with a getter that returns -1 when not
applicable. Every existing caller and test was rewritten to:
new LlmException(message, status, code, body, cause,
LlmException.ErrorType.X)
and instanceof checks were turned into:
e.getType() == LlmException.ErrorType.X
The class javadoc now shows a single try/catch + switch over
getType() as the recommended pattern.
2. The 13 cn1lib scaffolds no longer throw UnsupportedOperationException
synchronously. Each facade now uses the standard Codename One
NativeInterface pattern:
- A package-private Native<Feature> interface extending
NativeInterface defines the platform contract.
- The facade resolves it via NativeLookup.create(Native<Feature>.class)
in a background thread (Display.scheduleBackgroundTask) and
completes an AsyncResource on the EDT.
- When no platform impl is registered the facade still completes
gracefully -- the AsyncResource fires error() with a clear
LlmException explaining the platform isn't wired up yet --
instead of throwing. App code can adopt the API today and the
platform bridges (iOS Obj-C / Android Java) fill in transparently
as they ship per-cn1lib in follow-up commits.
Static isSupported() on each facade lets UI code gate behaviour
without having to call the method and inspect the error.
Versioning + release alignment (review note 3):
Each cn1lib pom inherits <parent> codenameone:8.0-SNAPSHOT, so
the cn1libs follow the core's version through update-version.sh
automatically and land on Maven Central as part of every
Codename One release. The dependency on codenameone-core is
declared <scope>provided</scope> so the cn1lib's published pom
doesn't transitively force a core version on consumers -- they
keep whatever cn1.version their app already declares.
Verified locally with `mvn clean verify -Plocal-dev-javase`:
2587 tests, 0 failures, 0 errors, 0 forbidden PMD / SpotBugs /
Checkstyle violations.
Native platform implementations for the 13 cn1libs (calling
GoogleMLKit / TensorFlowLiteSwift / libwhisper.a / Core ML / ONNX
Runtime on iOS and Android) require device-testable bindings and
ship in follow-up per-cn1lib commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matches the existing CodenameOne convention (58 sibling package-info files under com/codename1/*). Documents each package's purpose, the public surface (the single facade class), and the build-server's auto-injection behaviour. - com.codename1.ai -- main client / value-types / streaming surface - com.codename1.ai.mlkit.text / .barcode / .face / .labeling / .translate / .smartreply / .langid / .pose / .segmentation / .docscan - com.codename1.ai.tflite - com.codename1.ai.whisper - com.codename1.ai.imagegen (Stable Diffusion) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the previous stub-throwing single-module skeletons with proper multi-module cn1lib layouts. Each cn1-ai-* directory is now root + common + ios + android + javase + lib, with the .cn1lib mojo producing nativeios.zip / nativeand.zip / nativese.zip bundles. Native bridges call the real underlying SDKs (not stubs): - mlkit-text: GoogleMLKit/TextRecognition (iOS) + text-recognition - mlkit-barcode: GoogleMLKit/BarcodeScanning + barcode-scanning - mlkit-face: GoogleMLKit/FaceDetection + face-detection - mlkit-labeling: GoogleMLKit/ImageLabeling + image-labeling - mlkit-translate: GoogleMLKit/Translate + translate (with model d/l) - mlkit-smartreply: GoogleMLKit/SmartReply + smart-reply - mlkit-langid: GoogleMLKit/LanguageID + language-id - mlkit-pose: GoogleMLKit/PoseDetection + pose-detection - mlkit-segmentation: GoogleMLKit/SegmentationSelfie + segmentation-selfie - mlkit-docscan: VisionKit + CIDetector (iOS), play-services-mlkit-docscan - tflite: TensorFlowLiteObjC + org.tensorflow:tensorflow-lite - whisper: whisper.cpp C API + JNI .so (Android) - stablediffusion: Core ML / ONNX Runtime (local-build only; >2 GB) Generator + idempotency: scripts/gen-ai-cn1libs.py is the single source of truth -- the 13 cn1lib trees are 100% rewritten from it. CI runs the generator and fails if the checked-in tree drifts. JVM tests: Each common module ships a JUnit 5 test that exercises the facade contract against a mock NativeInterface. All 13 modules build and test cleanly via `mvn package`; .cn1lib archives include real native sources (verified by unzip). CI: - PR CI step packages all 13 cn1libs as part of the standard pipeline. - ai-cn1lib-native-check.yml lint-compiles every iOS Obj-C bridge via clang -fsyntax-only against stub ML Kit / TFLite headers. - ai-cn1lib-android-check.yml compiles every Android Java bridge with javac against android.jar + real ML Kit / TFLite / ONNX Runtime jars pulled from Google Maven, so missing symbols surface in PR review. Not yet wired up (separate follow-up): - End-to-end emulator/simulator UI integration tests that build a CN1 app per cn1lib, push it through ParparVM / Android Gradle, boot the iOS simulator / Android emulator, feed fixture images and assert on bridge output. The lint workflows above are a fast static gate but not behavioural; that work needs cn1app-archetype test scaffolding + Espresso/XCTest harnesses that aren't in scope here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iOS Obj-C cleanup: - Move whisper.cpp and Stable Diffusion C extern declarations OUTSIDE the @implementation block (was rejected by clang -- Obj-C only allows method definitions inside @implementation, not C functions / structs). - Forward-declare `struct whisper_context` and define `struct whisper_full_params` BEFORE the extern that takes it by value (was a forward-reference compile error). - Rebuild every other lib's .m file via the regenerated template so whitespace stays consistent (the previous output had mixed 8/0 leading indent from a textwrap.dedent quirk). Android lint workflow: - Replace fragile `yes | sdkmanager` (SIGPIPE under `set -o pipefail`) with printf'd y-stream. - Switch from a Gradle classpath probe to Maven dependency:copy with explicit `<type>aar</type>` entries, then unzip classes.jar out of each AAR. Most ML Kit deps are AAR-only on Google Maven, so the earlier classpath approach silently dropped half the symbols. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- iOS stub headers got included multiple times per source (each ML Kit SDK header symlinks to the same stub), producing duplicate-interface errors. Add `#pragma once` so clang dedupes. - The PR CI `git diff --quiet` check was tripping `set -e -o pipefail` in a way that made git emit its --help text instead of the actual diff (exit 129). Switch to `git status --porcelain` for the drift check, which is exit-code-safe under the same shell options. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#pragma once relies on canonical-path comparison. clang on macos-14 treats each of our 13 SDK-name symlinks as a distinct canonical path, so the same stub gets parsed many times -> duplicate-interface errors. Switching to #ifndef CN1_AI_STUB_HEADERS_INCLUDED / #define / #endif makes clang dedupe by content even when the file is reached through different symlinks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The earlier `Run JavaSE port unit tests` step already installs codenameone-core + codenameone-javase into the local m2 with the local-dev-javase profile, so the cn1-ai-* common modules can resolve codenameone-core from the repo without rebuilding javase (which needs the local-dev-javase profile and was failing on stub CEF classes when -am pulled it back into the reactor). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cn1lib build-mojo lives in codenameone-maven-plugin. Earlier PR steps only run its tests; they don't `install`, so subsequent steps can't resolve the plugin. Install it explicitly before the cn1-ai package step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The plugin depends on codenameone-designer (jar-with-dependencies), codenameone-android, codenameone-ios, codenameone-parparvm, and java-runtime. None of these are in m2 from earlier steps, so the install of just the plugin failed during resolution. Add -am to install the plugin's reactor dependencies in one shot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ules
Three fixes prompted by review of the prior PR:
1. Match `mvn cn1:generate-native-interfaces` output verbatim. The hand-
rolled iOS .h/.m had two real bugs:
- Whitespace: `- (NSString *)recognize:` instead of canonical
`-(NSString*)recognize:`. Cosmetic.
- **Wrong selectors for multi-param methods**: the prior generator
wrote `translate:(NSString*)param :(NSString*)param1 :(NSString*)
param2` (selector `translate:::`) instead of `translate:(NSString
*)param param1:(NSString*)param1 param2:(NSString*)param2`
(selector `translate:param1:param2:`). The CN1 runtime dispatcher
would NOT have found the multi-param methods.
Generator now produces output that diffs clean against
`cn1:generate-native-interfaces` for all 13 cn1libs.
2. Add the missing `javascript/` and `win/` modules. Every cn1lib now
has the canonical seven-module layout (root + common + ios +
android + javase + javascript + win + lib) and emits a `.cn1lib`
that bundles nativeios.zip + nativeand.zip + nativese.zip +
nativejavascript.zip + nativewin.zip.
3. Replace the stub-headers `clang -fsyntax-only` lint with a real
`pod install` + `xcodebuild build` per cn1lib. Each lib gets its
own job (parallel matrix) on macos-14: synthesise a static-library
Xcode project that links every `nativeios/*.m`, install the
matching `GoogleMLKit/*` / `TensorFlowLiteObjC` pod, and let
xcodebuild prove the bridges compile against the real SDK
headers. Catches what the stub-header lint silently missed --
notably the selector bug above.
JavaSE impls also now `implements NativeXxx` (NativeLookup on JavaSE
casts to the interface, so the missing `implements` clause would have
broken simulator runs once the JavaSE port wired this up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two screenshot tests under scripts/hellocodenameone/common that exercise the new AI UI surface added in this PR: - ChatViewScreenshotTest emits a representative system / user / assistant exchange with the typing indicator visible. Covers ChatBubbleUser, ChatBubbleAssistant, ChatBubbleSystem, and ChatTypingIndicator UIIDs against iOS Modern and Android Material themes via the existing build-ios / build-ios-metal / Build Android screenshot pipeline. - ChatInputScreenshotTest renders the input row standalone with all three optional buttons (attach, voice, send) visible -- baseline for ChatInput, ChatInputField, ChatSendButton, ChatAttachButton, and ChatVoiceButton. Both are registered in Cn1ssDeviceRunner.DEFAULT_TEST_CLASSES next to SheetScreenshotTest so they ride the same screenshot capture path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The synthesised CN1AIProbe.xcodeproj is too sparse for CocoaPods to inject HEADER_SEARCH_PATHS / FRAMEWORK_SEARCH_PATHS into via integrate_targets => true (the build configurations don't have the expected baseConfigurationReference slot, so pod install runs but the xcframework paths never reach xcodebuild). Result: every ML Kit import landed at "'MLKitTextRecognition/MLKTextRecognizer.h' file not found". Switch to integrate_targets => false. pod install now just downloads the pods + generates a per-target xcconfig under `Pods/Target Support Files/...`. xcodebuild then loads that xcconfig via -xcconfig so HEADER_SEARCH_PATHS / FRAMEWORK_SEARCH_PATHS reach the compiler. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bugs in the previous attempt: 1. `find Pods/Target\ Support\ Files -name '*.debug.xcconfig' | head -1` returned the first PER-POD xcconfig (PromisesObjC etc.) which only wires up that single pod's framework path. The build still couldn't find `MLKitTextRecognition/MLKTextRecognizer.h`. Explicitly target the aggregator `Pods-CN1AIProbe/Pods-CN1AIProbe.debug.xcconfig` which carries HEADER_SEARCH_PATHS / FRAMEWORK_SEARCH_PATHS for every transitive pod the consumer target depends on. 2. Empty bash array expansion under `set -u` exited the cn1libs that ship without a pod (whisper / stablediffusion / docscan) with "XCCONFIG_ARG[@]: unbound variable". Drop `set -u` from this step and switch to a plain string variable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three review-driven fixes:
1. Remove the win/ module from every cn1-ai-* lib. UWP is not a
runtime target -- the C# stubs were dead weight inside every
.cn1lib. Drop the module from the root pom, the lib pom, the
generator, and delete the existing win/ trees. .cn1lib archives
now ship 6 entries (was 7).
2. Replace `codename1.arg.ios.plistInject` build hints with implicit
simulator-side injection. Each JavaSE NativeXxxImpl constructor
calls a one-shot `ensureSimulatorHints()` that:
- reads `getProjectBuildHints()` (returns null off-simulator, so
this is a no-op on real devices)
- sets the default value ONLY when the hint is absent -- never
overwrites a developer-customised string
This way the developer never has to copy a Camera/Microphone
usage description out of docs into their properties file, but
stays in full control of the wording. Only mlkit-text, barcode,
and face declared `NSCameraUsageDescription` in the previous
plistInject form; same coverage, now via the helper.
3. Style ChatView in the modern themes. The earlier ChatView
screenshot rendered without bubble fills because the default
themes had no ChatBubble*/ChatInput*/ChatTypingIndicator
UIIDs. iOSModern now uses iMessage-style accent fills for user
bubbles + neutral fills for assistant bubbles + a centred muted
row for system; AndroidMaterial mirrors with Material 3 primary
container / surface variant tokens. ChatView itself now wraps
each bubble in a FlowLayout row (USER -> RIGHT, ASSISTANT ->
LEFT, SYSTEM -> CENTER) so the bubble's visible width tracks
its content rather than spanning the chat surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pod-generated xcconfig lives under `Pods/Target Support Files/Pods-CN1AIProbe/...` -- a path with a space. Expanding it through an unquoted `$XCCONFIG_ARG` made the shell word-split it, so xcodebuild saw `-xcconfig Pods/Target` followed by `Support` as a positional arg -> "Unknown build action 'Support'" exit 65. Switch to a bash array, quote the expansion, and rely on the already-dropped `set -u` so an empty array doesn't trip up the no-pod cn1libs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous synth pbxproj + `:integrate_targets => false` approach fell over on ML Kit because ML Kit ships as `.xcframework` (multi-arch wrapper) -- not raw `.framework`. CocoaPods normally extracts the correct simulator slice via a script phase added during integration, but with `integrate_targets => false` we skipped that phase, so the framework lookup `#import <MLKitTextRecognition/MLKTextRecognizer.h>` saw the wrapper directory and not the actual framework. Switch to xcodegen (brew on macos-14) to produce a real, complete pbxproj that CocoaPods can integrate normally. With normal pod install the workspace gets: - baseConfigurationReference injected on each build configuration - the [CP] Embed Pods Frameworks script phase - the xcframework -> framework extraction phase xcodebuild now builds via `-workspace CN1AIProbe.xcworkspace` so the embed phase + extracted frameworks reach clang's framework search path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
xcodegen 2.45.4 emits projects in Xcode 16's objectVersion=77 format. The macos-14 runner ships Xcode 15.4, which refused them with "future Xcode project file format (77)". macos-15 carries Xcode 16+ so it can read the project. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
macos-15 runners returned 403 on actions/checkout for this repo (org runner policy, presumably). Revert to macos-14 and explicitly select the highest-version Xcode 16.* installed on the runner image so the default xcode-select (Xcode 15.4) doesn't reject xcodegen's objectVersion=77 projects. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two iterations on the matrix: 1. The per-class ML Kit header paths I was using (`<MLKitTextRecognition/MLKTextRecognizer.h>` etc.) aren't actually exported by the GoogleMLKit pods. ML Kit ships ONE umbrella header per framework, named after the framework itself. Switch every iOS bridge to import the umbrella (`<MLKitTextRecognition/MLKitText Recognition.h>`, `<MLKitVision/MLKitVision.h>`, etc.) so clang's framework header search resolves and the public classes -- which the umbrella re-exports -- stay visible to the .m bodies. 2. The cn1libs that ship without a CocoaPod (whisper / stable diffusion) declare extern symbols that the real build server resolves against the bundled static library / Swift runner. The probe build had no such library so the link phase failed. Switch the probe target from `framework` to `library.static` -- .a archives don't link, they archive .o files, so unresolved externs are fine. We still compile-check every source file, which is the whole point of this gate. 3. Also fix tflite's umbrella import: the pod is `TensorFlowLiteObjC` but the produced framework's module name -- the one used in the angle-bracket prefix -- is `TFLTensorFlowLite`. So `<TFLTensorFlowLite/TFLTensorFlowLite.h>`, not `<TensorFlowLiteObjC/...>`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ML Kit splits each framework into a user-facing recognizer/detector module + a sibling Common module holding the actual class / property definitions. clang's modules system requires importing the Common one explicitly even though the main framework re-exports those symbols at link time. Add the Common imports for the four cn1libs that hit this: - mlkit-text: MLKitTextRecognitionCommon - mlkit-labeling: MLKitImageLabelingCommon - mlkit-pose: MLKitPoseDetectionCommon - mlkit-segmentation: MLKitSegmentationCommon Also fix mlkit-text's recognizer-options class name -- the canonical options are `MLKTextRecognizerOptions`, not the `MLKCommonTextRecognizer Options` placeholder I had. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MLKSegmentationMask exposes only `pixelBuffer` (a CVPixelBuffer); the mask's width / height come from CVPixelBufferGetWidth / GetHeight, not properties on the mask itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MLKSegmentationMask's CVPixelBuffer property is `buffer`, not `pixelBuffer`. My previous switch to `.pixelBuffer` was based on a faulty memory of the API. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces the full AI/LLM surface for Codename One plus the
build-server plumbing that auto-wires every native dependency it
needs. Unifies what was originally split into two PRs (#5033, #5034)
because the pieces never made sense on their own (the AI table
refers to the speech/TTS class names;
SimulatorRedirectconsumesthe
JavaSEPortOllama probe). Rebased on current master.com.codename1.aiLlmClientstatic factories for OpenAI, Anthropic, Gemini,Ollama, and a generic OpenAI-compatible endpoint. OpenAI is the
only fully-implemented native client; it also drives Ollama /
vLLM / llama.cpp because the wire format matches. Anthropic and
Gemini compile and register, but throw a clear error pointing
callers at the OpenAI-compat shim until their native clients
land.
ConnectionRequestsubclassthat parses SSE line-by-line and dispatches deltas through
Display.callSerially.AsyncResource.cancel()kills thesocket.
ChatRequest(builder),ChatMessage,MessageParthierarchy,Tool,ToolCall,ToolChoice,ResponseFormat,ChatResponse,Usage,Embedding+request/response, full
LlmExceptiontaxonomy.ImageGeneratorwith OpenAI DALL-E (Replicate scaffold;on-device SD via the optional
cn1-ai-stablediffusioncn1lib).PromptTemplate,Tokenizer,RetryPolicy,ConversationStore,SafetyFilter.com.codename1.media.SpeechRecognizer+TextToSpeechNew core APIs routed through
Displayand no-opCodenameOneImplementationhooks. Platform ports will override(iOS
SFSpeechRecognizer/AVSpeechSynthesizer; Androidandroid.speech.*).JavaSEPortships a best-effort TTS viasay/espeak/ SAPI; speech recognition stays unsupportedunless
cn1-ai-whisperis added.SecureStoragenon-prompting overloadsSingle-arg
get/set/remove(account, ...)added next to theexisting biometric-gated methods. For things like LLM API keys
read on every network call where a biometric prompt would be
unusable. Base class returns
null/falseon unsupported ports.com.codename1.components.ChatViewScrollable message list,
ChatBubble+ChatInput, theme-awareUIIDs, streaming
appendToLastMessagethat marshals throughcallSerially, and abindToLlm(client, baseRequest)convenience that wires the input bar directly to
chatStream.Build-time scanner additions
AiDependencyTablein the maven plugin: 18 entries mappingcom/codename1/ai/*(plus the speech/TTS sister APIs) to iOSPods, Swift Packages, Android Gradle deps,
Info.plistusagestrings, and Android permissions.
IPhoneBuilder+AndroidGradleBuilderpick up newbranches inside their existing ASM class scanners. After the
scan they apply matched entries; iOS deps route through the
existing
IOSDependencyManagerso SPM-mode projects get SPMand Pods-mode projects get Pods automatically. Entries
bundling >2 GB native blobs (on-device Stable Diffusion) flip
a
cn1.ai.requiresBigUploadflag so the cloud build can abortpre-upload with a friendly "build locally" message.
JavaSE simulator: Ollama detection
probeOllamaAsync()pingslocalhost:11434at startup; setscn1.ai.ollamaDetectedwhen reachable.SimulatorRedirectreads that property and, with
cn1.ai.simulatorRedirect=autoor
=ollama, routes anyLlmClient.openai(...)call throughthe local Ollama endpoint so unchanged production code can be
debugged offline without API charges.
Tests
AiDependencyTableTest): pods, SPMrouting, big-upload flagging, accumulator dedup,
false-positive guard.
ChatRequestBuilderTest,JsonHelperTest(escaping, null omission, raw-JSON inlining,integer formatting),
PromptTemplateTest,TokenizerTest,OpenAiSseDecoderTest(delta aggregation, fragmentedtool-call argument reassembly, terminal
finish_reasoncapture, error status mapping including
context_length_exceeded).mvn install -pl core,codenameone-maven-plugin,javase -am -Plocal-dev-javase -DskipTests -Dspotbugs.skip=truebuilds clean.scripts/create-ai-cn1lib.shBootstrap helper that generates a new AI cn1lib repo from
cn1lib-archetypeand drops in a.github/workflows/publish.ymlthat publishes to Maven Central on every merge to master.
Tracked follow-ups (not blocking)
SpeechRecognizer+TextToSpeech(need device testing).cn1-ai-*cn1lib repos themselves (bootstrap via the new script).Speech.framework/AVFAudio/CoreMLetc. — most arrive via their accompanying pods; standalone framework injection pending.Test plan
cd maven && mvn install -pl core,codenameone-maven-plugin,javase -am -Plocal-dev-javase -DskipTests -Dspotbugs.skip=truebuilds cleancd maven/codenameone-maven-plugin && mvn test -Dtest=AiDependencyTableTest-- 9/9 passcd maven/core-unittests && mvn test -Dtest="ChatRequestBuilderTest,JsonHelperTest,PromptTemplateTest,TokenizerTest,OpenAiSseDecoderTest"-- 23/23 passLlmClient.openai(System.getenv("OPENAI_API_KEY")).chatStream(req, listener)streams tokenscn1.ai.simulatorRedirect=ollamaroutesLlmClient.openai("sk-fake")calls tolocalhost:11434TextToSpeech.speak("hello")works on the simulator (macOS viasay)android.xpermissionswith CAMERA + using a (future)com.codename1.ai.mlkit.barcode.*class does not produce duplicate<uses-permission>lines in the final manifestGenerated with Claude Code