Machine learning based anomaly and attack detection is a topic under the spotlights for more than one decade. It is raising a significant research effort worldwide. The reasons of this interest lie in the ability of machine learning to introduce a large part of autonomy in the attacks detection process compared to the classical signature based approach, and then reduces the management cost of networks for NetOps, provides rapid and efficient results, and is prone to detect 0d attacks. This paper confirms this efficiency showing that using the simple Random Forest (RF) algorithms together with features selection allows detection ratio greater than 99%. However, despite a large set of attempts on the training stage for improving this detection ratio, RF never detects 100% of attacks. It especially always misses some attacks in the traffic, especially under-represented ones in the training dataset. False Negatives are certainly the most dramatic events for network security. In addition, it makes adversarial learning very easy to perform. This paper then presents a controversial study on machine learning based attack detection (using the significantly illustrative RF example), and its trustworthiness limits. This paper is then trying to break the current dogma that machine learning is THE solution for future cyber-security systems.