20170208-NOOR

Experiment design

Revision of networks of ontologies comparing delete/replace/add and new reference (4 agents; 10000 iterations)

Hypothesis: The observation of 20150218-NOOR still hold in the improved implementation

Experimental setting: Repetition of 20150218-NOOR with:

10000 iterations
new reference alignments (not limited to leaves)
new incoherence measure (Alcomo: considering subsumption correspondences)

(Note: LogMap, Alcomo and references have been separated in 20170208b-NOOR

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 20170208

Lazy lavender hash: 738f55c3e7dc6fc16ebcfe070adc314dd4703d21

Parameters: params.sh

Command line (script.sh):

java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=delete -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec

java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=replace -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec

java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Raw results

20170208-NOOR-4-10000-add.tsv 20170208-NOOR-4-10000-add.txt 20170208-NOOR-4-10000-del.tsv 20170208-NOOR-4-10000-del.txt 20170208-NOOR-4-10000-repl.tsv 20170208-NOOR-4-10000-repl.txt

Result exploration

modality	size	success	convergence	Incoherence degree	Precision	F-measure	Recall
delete	6	.99	310	0.	1.	.13	.07
replace	11	.99	1076	.01	.96	.23	.13
add	33	.98	1235	.16	.77	.53	.41

In red the average success rate (over 10 runs) of delete;
In blue the average success rate (over 10 runs) of replace;
In green the average success rate (over 10 runs) of add;
In dashed, the semantic F-measure.

Above: semantic precision;
Below: semantic recall.

(drawn from data scaled every 100 measures)

Analysis

Key points:

The results in terms of F-measure for the adaptation operators beyond DELETE are far higher than in 20150218-NOOR. This is due to the new (corrected) reference alignments
DELETE and REPLACE have now different F-measure. This is due to this new F-measure.
REPLACE and ADD end up with non-zero incoherence measure. This is explained by the the shadowing mechanism. It was undetected previously due to the way Alcomo's incoherence measure was parameterised to only take into account "=" correspondences.

Further experiments:

This file can be retrieved from URL https://sake.re/20170208-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20170208-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)